Authentication of objects using steganography

ABSTRACT

Steganographic methods for encoding auxiliary data into a host signal are used to authenticate physical and electronic objects. One claim recites an object comprising: a first graphic provided on an object surface, and a storage device carried by the object. The storage device includes a second graphic stored therein. The first and second graphics are related graphics, and the second graphic includes first steganographic encoding including a plural-bit payload. Of course, other combinations are provided as well.

RELATED APPLICATION DATA

This application is a continuation of U.S. patent application Ser. No.10/992,614, filed Nov. 17, 2004 (published as U.S. 2005-0094848 A1),which is a division of U.S. Pat. No. 09/837,564, filed Apr. 17, 2001(published as U.S. 2002-0009208 A1), which claims the benefit of U.S.Provisional Patent Application No. 60/198,849, filed Apr. 21, 2000.

The Ser. No. 09/837,564 application is also related to U.S. patentapplication Ser. No. 09/503,881, filed Feb. 14, 2000, which is acontinuation in part of application Ser. No. 09/186,962, filed Nov. 5,1998, which is a continuation of application Ser. No. 08/649,419, filedMay 16, 1996, (now U.S. Pat. No. 5,862,260), which is a continuation inpart of application Ser. No. 08/637,531, filed Apr. 25, 1996, (now U.S.Pat. No. 5,822,436), which is a continuation in part of application Ser.No. 08/512,993 filed Aug. 9, 1995, (abandoned in favor of FWC Ser. No.08/763,847, now issued as U.S. Pat. No. 5,841,886). The Ser. No.09/837,564 application also claims priority to U.S. Provisional patentapplication 60/198,138 filed Apr. 17, 2000.

Each of the above patent documents is hereby incorporated by reference.

TECHNICAL FIELD

The invention relates to encoding and decoding an auxiliary signal intoa media signal (e.g., image, audio and video), such that the auxiliarysignal is substantially imperceptible in the resulting signal, yetrecoverable by an automated decoding process.

BACKGROUND AND SUMMARY

Digital watermarking is a process for modifying media content to embed amachine-readable code into the data content. The data may be modifiedsuch that the embedded code is imperceptible or nearly imperceptible tothe user, yet may be detected through an automated detection process.Most commonly, digital watermarking is applied to media such as images,audio signals, and video signals. However, it may also be applied toother types of data, including documents (e.g., through line, word orcharacter shifting), software, multi-dimensional graphics models, andsurface textures of objects.

Digital watermarking systems have two primary components: an embeddingcomponent that embeds the watermark in the media content, and a readingcomponent that detects and reads the embedded watermark. The embeddingcomponent embeds a watermark pattern by altering data samples of themedia content. The reading component analyzes content to detect whethera watermark pattern is present. In applications where the watermarkencodes information, the reader extracts this information from thedetected watermark.

One challenge to the developers of watermark embedding and readingsystems is to ensure that the watermark is detectable even if thewatermarked media content is transformed in some fashion. The watermarkmay be corrupted intentionally, so as to bypass its copy protection oranti-counterfeiting functions, or unintentionally through varioustransformations that result from routine manipulation of the content. Inthe case of watermarked images, such manipulation of the image maydistort the watermark pattern embedded in the image.

The invention provides watermark structures, and related embedders,detectors, and readers for processing the watermark structures. Inaddition, it provides a variety of methods and applications associatedwith the watermark structures, embedders, detectors and readers. Whileadapted for images, the watermark system applies to other electronic andphysical media. For example, it can be applied to electronic objects,including image, audio and video signals. It can be applied to markblank paper, film and other substrates, and it can be applied bytexturing object surfaces for a variety of applications, such asidentification, authentication, etc. The detector and reader can operateon a signal captured from a physical object, even if that capturedsignal is distorted.

The watermark structure can have multiple components, each havingdifferent attributes. To name a few, these attributes include function,signal intensity, transform domain of watermark definition (e.g.,temporal, spatial, frequency, etc.), location or orientation in hostsignal, redundancy, level of security (e.g., encrypted or scrambled).When describing a watermark signal in the context of this document,intensity refers to an embedding level while strength describes readinglevel (though the terms are sometimes used interchangeably). Thecomponents of the watermark structure may perform the same or differentfunctions. For example, one component may carry a message, while anothercomponent may serve to identify the location or orientation of thewatermark in a combined signal. Moreover, different messages may beencoded in different temporal or spatial portions of the host signal,such as different locations in an image or different time frames ofaudio or video.

Watermark components may have different signal intensities. For example,one component may carry a longer message, yet have smaller signalintensity than another component, or vice-versa. The embedder may adjustthe signal intensity by encoding one component more redundantly thanothers, or by applying a different gain to the components. Additionally,watermark components may be defined in different transform domains. Onemay be defined in a frequency domain, while another may be defined in aspatial or temporal domain.

The watermark components may be located in different spatial or temporallocations in the host signal. In images, for example, differentcomponents may be located in different parts of the image. Eachcomponent may carry a different message or perform a different function.In audio or video, different components may be located in different timeframes of the signal.

The watermark components may be defined, embedded and extracted indifferent domains. Examples of domains include spatial, temporal andfrequency domains. A watermark may be defined in a domain by specifyinghow it alters the host signal in that domain to effect the encoding ofthe watermark component. A frequency domain component alters the signalin the frequency domain, while a spatial domain component alters thesignal in the spatial domain. Of course, such alterations may have animpact that extends across many transform domains.

While described here as watermark components, one can also construe thecomponents to be different watermarks. This enables the watermarktechnology described throughout this document to be used in applicationsusing two or more watermarks. For example, some copy protectionapplications of the watermark structure may use two or more watermarks,each performing similar or different function. One mark may be morefragile than another, and thus, disappear when the combined signal iscorrupted or transformed in some fashion. The presence or lack of awatermark or watermark component conveys information to the detector toinitiate or prohibit some action, such as playback, copying or recordingof the marked signal.

A watermark system may include an embedder, detector, and reader. Thewatermark embedder encodes a watermark signal in a host signal to createa combined signal. The detector looks for the watermark signal in apotentially corrupted version of the combined signal, and computes itsorientation. Finally, a reader extracts a message in the watermarksignal from the combined signal using the orientation to approximate theoriginal state of the combined signal.

There are a variety of alternative embodiments of the embedder anddetector. One embodiment of the embedder performs error correctioncoding of a binary message, and then combines the binary message with acarrier signal to create a component of a watermark signal. It thencombines the watermark signal with a host signal. To facilitatedetection, it may also add a detection component to form a compositewatermark signal having a message and detection component. The messagecomponent includes known or signature bits to facilitate detection, andthus, serves a dual function of identifying the mark and conveying amessage. The detection component is designed to identify the orientationof the watermark in the combined signal, but may carry an informationsignal as well. For example, the signal values at selected locations inthe detection component can be altered to encode a message.

One embodiment of the detector estimates an initial orientation of awatermark signal in the multidimensional signal, and refines the initialorientation to compute a refined orientation. As part of the process ofrefining the orientation, this detector computes at least oneorientation parameter that increases correlation between the watermarksignal and the multidimensional signal when the watermark ormultidimensional signal is adjusted with the refined orientation.

Another detector embodiment computes orientation parameter candidates ofa watermark signal in different portions of the target signal, andcompares the similarity of orientation parameter candidates from thedifferent portions. Based on this comparison, it determines whichcandidates are more likely to correspond to a valid watermark signal.Yet another detector embodiment estimates orientation of the watermarkin a target signal suspected of having a watermark. The detector thenuses the orientation to extract a measure of the watermark in thetarget. It uses the measure of the watermark to assess merits of theestimated orientation. In one implementation, the measure of thewatermark is the extent to which message bits read from the targetsignal match with expected bits. Another measure is the extent to whichvalues of the target signal are consistent with the watermark signal.The measure of the watermark signal provides information about themerits of a given orientation that can be used to find a better estimateof the orientation.

Another aspect of the invention is a method for encoding auxiliary datainto a host signal. This method computes a content specific messagedependent on the host signal, encodes the content specific message intoa watermark signal, and embeds the watermark in the host signal suchthat the watermark signal is substantially imperceptible in the hostsignal. One specific implementation embeds data representing salientfeatures of the host signal into the watermark. For example, for photoIDs, the method embeds the spatial location of salient features of thephoto into the watermark.

Another implementation computes a semi-sensitive hash of the hostsignal, such as a low pass filtering of the signal, and embeds the hashinto the watermark.

Another aspect of the invention is a method for authenticating a mediaobject. This method transforms a media signal to a frequency domaincomprising an array of frequency coefficients. It selects a first set offrequency coefficients, and alters the selected first set of frequencycoefficients so that values of the coefficients in the set correspond toa pattern. The pattern of the media signal is authenticated by comparinga pattern of the values of the frequency coefficients in the set with anexpected pattern.

In one implementation, pairs of coefficients are modified to encode adelta pattern. For each coefficient in the first set, the method selectsa corresponding coefficient in a corresponding set, and changes valuesof the coefficients in the first set and the corresponding coefficientsin the corresponding set such that the differences between thecoefficient values in the first set and the corresponding set form adesired delta pattern. This delta pattern may be a pseudorandom numbergenerated from a key number. The key number may be derived from themedia signal, such as a hash of the media signal. Alternatively, it maybe a user or object identifier number associated with the media signal.

One combination according to the invention is an object comprising: afirst graphic including steganographic data hidden therein and providedon the object; a copy of the first graphic including steganographic datahidden therein and provided on the object or carried by the object, andthe steganographic data hidden in the first graphic and the copy isrelated in a manner that enables validity of the object to be verified.

Another combination according to the invention is an object comprising:a first graphic, wherein the first graphic comprises firststeganographic encoding therein; and electronic or integrated circuitry,wherein the electronic or integrated circuitry comprises a secondgraphic stored therein, wherein the second graphic comprises secondsteganographic encoding, and wherein the second encoding comprises aplural-bit identifier.

Still another combination is an object comprising: a first graphicprovided on an object surface; and a storage device carried by theobject, wherein the storage device comprises a second graphic storedtherein, the first and second graphic are related graphics, and whereinthe second graphic comprises first steganographic encoding including aplural-bit payload.

Further features, embodiments and combinations of the invention willbecome apparent from the following detailed description and accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an image watermark system.

FIG. 2 is a block diagram illustrating an image watermark embedder.

FIG. 3 is a spatial frequency domain plot of a detection watermarksignal.

FIG. 4 is a flow diagram of a process for detecting a watermark signalin an image and computing its orientation within the image.

FIG. 5 is a flow diagram of a process reading a message encoded in awatermark.

FIG. 6 is a diagram depicting an example of a watermark detectionprocess.

FIG. 7 is a diagram depicting the orientation of a transformed imagesuperimposed over the original orientation of the image at the time ofwatermark encoding.

FIG. 8 is a diagram illustrating an implementation of a watermarkembedder.

FIG. 9 is a diagram depicting an assignment map used to map raw bits ina message to locations within a host image.

FIG. 10 illustrates an example of a watermark orientation signal in aspatial frequency domain.

FIG. 11 illustrates the orientation signal shown in FIG. 10 in thespatial domain.

FIG. 12 is a diagram illustrating an overview of a watermark detectorimplementation.

FIG. 13 is a diagram illustrating an implementation of the detectorpre-processor depicted generally in FIG. 12.

FIG. 14 is a diagram illustrating a process for estimating rotation andscale vectors of a detection watermark signal.

FIG. 15 is a diagram illustrating a process for refining the rotationand scale vectors, and for estimating differential scale parameters ofthe detection watermark signal.

FIG. 16 is a diagram illustrating a process for aggregating evidence ofthe orientation signal and orientation parameter candidates from two ormore frames.

FIG. 17 is a diagram illustrating a process for estimating translationparameters of the detection watermark signal.

FIG. 18 is a diagram illustrating a process for refining orientationparameters using known message bits in the watermark message.

FIG. 19 is a diagram illustrating a process for reading a watermarkmessage from an image, after re-orienting the image data using anorientation vector.

FIG. 20 is a diagram of a computer system that serves as an operatingenvironment for software implementations of a watermark embedder,detector and reader.

FIG. 21 is a diagram illustrating aspects of a watermark embeddingprocess where sets of frequency coefficients are adjusted so that thesets have a predetermined relationship.

DETAILED DESCRIPTION

1.0 Introduction

A watermark can be viewed as an information signal that is embedded in ahost signal, such as an image, audio, or some other media content.Watermarking systems based on the following detailed description mayinclude the following components: 1) An embedder that inserts awatermark signal in the host signal to form a combined signal; 2) Adetector that determines the presence and orientation of a watermark ina potentially corrupted version of the combined signal; and 3) A readerthat extracts a watermark message from the combined signal. In someimplementations, the detector and reader are combined.

The structure and complexity of the watermark signal can varysignificantly, depending on the application. For example, the watermarkmay be comprised of one or more signal components, each defined in thesame or different domains. Each component may perform one or morefunctions. Two primary functions include acting as an identifier tofacilitate detection and acting as an information carrier to convey amessage. In addition, components may be located in different spatial ortemporal portions of the host signal, and may carry the same ordifferent messages.

The host signal can vary as well. The host is typically some form ofmulti-dimensional media signal, such as an image, audio sequence orvideo sequence. In the digital domain, each of these media types isrepresented as a multi-dimensional array of discrete samples. Forexample, a color image has spatial dimensions (e.g., its horizontal andvertical components), and color space dimensions (e.g., YUV or RGB).Some signals, like video, have spatial and temporal dimensions.Depending on the needs of a particular application, the embedder mayinsert a watermark signal that exists in one or more of thesedimensions.

In the design of the watermark and its components, developers are facedwith several design issues such as: the extent to which the mark isimpervious to jamming and manipulation (either intentional orunintentional); the extent of imperceptibility; the quantity ofinformation content; the extent to which the mark facilitates detectionand recovery, and the extent to which the information content can berecovered accurately.

For certain applications, such as copy protection or authentication, thewatermark should be difficult to tamper with or remove by those seekingto circumvent it. To be robust, the watermark must withstand routinemanipulation, such as data compression, copying, linear transformation,flipping, inversion, etc., and intentional manipulation intended toremove the mark or make it undetectable. Some applications require thewatermark signal to remain robust through digital to analog conversion(e.g., printing an image or playing music), and analog to digitalconversion (e.g., scanning the image or digitally sampling the music).In some cases, it is beneficial for the watermarking technique towithstand repeated watermarking.

A variety of signal processing techniques may be applied to address someor all of these design considerations. One such technique is referred toas spreading. Sometimes categorized as a spread spectrum technique,spreading is a way to distribute a message into a number of components(chips), which together make up the entire message. Spreading makes themark more impervious to jamming and manipulation, and makes it lessperceptible.

Another category of signal processing technique is error correction anddetection coding. Error correction coding is useful to reconstruct themessage accurately from the watermark signal. Error detection codingenables the decoder to determine when the extracted message has anerror.

Another signal processing technique that is useful in watermark codingis called scattering. Scattering is a method of distributing the messageor its components among an array of locations in a particular transformdomain, such as a spatial domain or a spatial frequency domain. Likespreading, scattering makes the watermark less perceptible and moreimpervious to manipulation.

Yet another signal processing technique is gain control. Gain control isused to adjust the intensity of the watermark signal. The intensity ofthe signal impacts a number of aspects of watermark coding, includingits perceptibility to the ordinary observer, and the ability to detectthe mark and accurately recover the message from it.

Gain control can impact the various functions and components of thewatermark differently. Thus, in some cases, it is useful to control thegain while taking into account its impact on the message and orientationfunctions of the watermark or its components. For example, in awatermark system described below, the embedder calculates a differentgain for orientation and message components of an image watermark.

Another useful tool in watermark embedding and reading is perceptualanalysis. Perceptual analysis refers generally to techniques forevaluating signal properties based on the extent to which thoseproperties are (or are likely to be) perceptible to humans (e.g.,listeners or viewers of the media content). A watermark embedder cantake advantage of a Human Visual System (HVS) model to determine whereto place a watermark and how to control the intensity of the watermarkso that chances of accurately recovering the watermark are enhanced,resistance to tampering is increased, and perceptibility of thewatermark is reduced. Such perceptual analysis can play an integral rolein gain control because it helps indicate how the gain can be adjustedrelative to the impact on the perceptibility of the mark. Perceptualanalysis can also play an integral role in locating the watermark in ahost signal. For example, one might design the embedder to hide awatermark in portions of a host signal that are more likely to mask themark from human perception.

Various forms of statistical analyses may be performed on a signal toidentify places to locate the watermark, and to identify places where toextract the watermark. For example, a statistical analysis can identifyportions of a host image that have noise-like properties that are likelyto make recovery of the watermark signal difficult. Similarly,statistical analyses may be used to characterize the host signal todetermine where to locate the watermark.

Each of the techniques may be used alone, in various combinations, andin combination with other signal processing techniques.

In addition to selecting the appropriate signal processing techniques,the developer is faced with other design considerations. Oneconsideration is the nature and format of the media content. In the caseof digital images, for example, the image data is typically representedas an array of image samples. Color images are represented as an arrayof color vectors in a color space, such as RGB or YUV. The watermark maybe embedded in one or more of the color components of an image. In someimplementations, the embedder may transform the input image into atarget color space, and then proceed with the embedding process in thatcolor space.

2.0 Digital Watermark Embedder and Reader Overview

The following sections describe implementations of a watermark embedderand reader that operate on digital signals. The embedder encodes amessage into a digital signal by modifying its sample values such thatthe message is imperceptible to the ordinary observer in output form. Toextract the message, the reader captures a representation of the signalsuspected of containing a watermark and then processes it to detect thewatermark and decode the message.

FIG. 1 is a block diagram summarizing signal processing operationsinvolved in embedding and reading a watermark. There are three primaryinputs to the embedding process: the original, digitized signal 100, themessage 102, and a series of control parameters 104. The controlparameters may include one or more keys. One key or set of keys may beused to encrypt the message. Another key or set of keys may be used tocontrol the generation of a watermark carrier signal or a mapping ofinformation bits in the message to positions in a watermark informationsignal.

The carrier signal or mapping of the message to the host signal may beencrypted as well. Such encryption may increase security by varying thecarrier or mapping for different components of the watermark orwatermark message. Similarly, if the watermark or watermark message isredundantly encoded throughout the host signal, one or more encryptionkeys can be used to scramble the carrier or signal mapping for eachinstance of the redundantly encoded watermark. This use of encryptionprovides one way to vary the encoding of each instance of theredundantly encoded message in the host signal. Other parameters mayinclude control bits added to the message, and watermark signalattributes (e.g., orientation or other detection patterns) used toassist in the detection of the watermark.

Apart from encrypting or scrambling the carrier and mapping information,the embedder may apply different, and possibly unique carrier or mappingfor different components of a message, for different messages, or fromdifferent watermarks or watermark components to be embedded in the hostsignal. For example, one watermark may be encoded in a block of sampleswith one carrier, while another, possibly different watermark, isencoded in a different block with a different carrier. A similarapproached to use different mappings in different blocks of the hostsignal.

The watermark embedding process 106 converts the message to a watermarkinformation signal. It then combines this signal with the input signaland possibly another signal (e.g., an orientation pattern) to create awatermarked signal 108. The process of combining the watermark with theinput signal may be a linear or non-linear function. Examples ofwatermarking functions include: S*=S+gX; S*=S(1+gX); and S*=S e^(gx);where S* is the watermarked signal vector, S is the input signal vector,and g is a function controlling watermark intensity. The watermark maybe applied by modulating signal samples S in the spatial, temporal orsome other transform domain.

To encode a message, the watermark encoder analyzes and selectivelyadjusts the host signal to give it attributes that correspond to thedesired message symbol or symbols to be encoded. There are many signalattributes that may encode a message symbol, such as a positive ornegative polarity of signal samples or a set of samples, a given parity(odd or even), a given difference value or polarity of the differencebetween signal samples (e.g., a difference between selected spatialintensity values or transform coefficients), a given distance valuebetween watermarks, a given phase or phase offset between differentwatermark components, a modulation of the phase of the host signal, amodulation of frequency coefficients of the host signal, a givenfrequency pattern, a given quantizer (e.g., in Quantization IndexModulation) etc.

Some processes for combining the watermark with the input signal aretermed non-linear, such as processes that employ dither modulation,modify least significant bits, or apply quantization index modulation.One type of non-linear modulation is where the embedder sets signalvalues so that they have some desired value or characteristiccorresponding to a message symbol. For example, the embedder maydesignate that a portion of the host signal is to encode a given bitvalue. It then evaluates a signal value or set of values in that portionto determine whether they have the attribute corresponding to themessage bit to be encoded. Some examples of attributes include apositive or negative polarity, a value that is odd or even, a checksum,etc. For example, a bit value may be encoded as a one or zero byquantizing the value of a selected sample to be even or odd. As anotherexample, the embedder might compute a checksum or parity of an N bitpixel value or transform coefficient and then set the least significantbit to the value of the checksum or parity. Of course, if the signalalready corresponds to the desired message bit value, it need not bealtered. The same approach can be extended to a set of signal sampleswhere some attribute of the set is adjusted as necessary to encode adesired message symbol. These techniques can be applied to signalsamples in a transform domain (e.g., transform coefficients) or samplesin the temporal or spatial domains.

Quantization index modulation techniques employ a set of quantizers. Inthese techniques, the message to be transmitted is used as an index forquantizer selection. In the decoding process, a distance metric isevaluated for all quantizers and the index with the smallest distanceidentifies the message value.

The watermark detector 110 operates on a digitized signal suspected ofcontaining a watermark. As depicted generally in FIG. 1, the suspectsignal may undergo various transformations 112, such as conversion toand from an analog domain, cropping, copying, editing,compression/decompression, transmission etc. Using parameters 114 fromthe embedder (e.g., orientation pattern, control bits, key(s)), itperforms a series of correlation or other operations on the capturedimage to detect the presence of a watermark. If it finds a watermark, itdetermines its orientation within the suspect signal.

Using the orientation, if necessary, the reader 116 extracts themessage. Some implementations do not perform correlation, but instead,use some other detection process or proceed directly to extract thewatermark signal. For instance in some applications, a reader may beinvoked one or more times at various temporal or spatial locations in anattempt to read the watermark, without a separate pre-processing stageto detect the watermark's orientation.

Some implementations require the original, un-watermarked signal todecode a watermark message, while others do not. In those approacheswhere the original signal is not necessary, the original un-watermarkedsignal can still be used to improve the accuracy of message recovery.For example, the original signal can be removed, leaving a residualsignal from which the watermark message is recovered. If the decoderdoes not have the original signal, it can still attempt to removeportions of it (e.g., by filtering) that are expected not to contain thewatermark signal.

Watermark decoder implementations use known relationships between awatermark signal and a message symbol to extract estimates of messagesymbol values from a signal suspected of containing a watermark. Thedecoder has knowledge of the properties of message symbols and how andwhere they are encoded into the host signal to encode a message. Forexample, it knows how message bit values of one and a zero are encodedand it knows where these message bits are originally encoded. Based onthis information, it can look for the message properties in thewatermarked signal. For example, it can test the watermarked signal tosee if it has attributes of each message symbol (e.g., a one or zero) ata particular location and generate a probability measure as an indicatorof the likelihood that a message symbol has been encoded. Knowing theapproximate location of the watermark in the watermarked signal, thereader implementation may compare known message properties with theproperties of the watermarked signal to estimate message values, even ifthe original signal is unavailable. Distortions to the watermarkedsignal and the host signal itself make the watermark difficult torecover, but accurate recovery of the message can be enhanced using avariety of techniques, such as error correction coding, watermark signalprediction, redundant message encoding, etc.

One way to recover a message value from a watermarked signal is toperform correlation between the known message property of each messagesymbol and the watermarked signal. If the amount of correlation exceedsa threshold, for example, then the watermarked signal may be assumed tocontain the message symbol. The same process can be repeated fordifferent symbols at various locations to extract a message. A symbol(e.g., a binary value of one or zero) or set of symbols may be encodedredundantly to enhance message recovery.

In some cases, it is useful to filter the watermarked signal to removeaspects of the signal that are unlikely to be helpful in recovering themessage and/or are likely to interfere with the watermark message. Forexample, the decoder can filter out portions of the original signal andanother watermark signal or signals. In addition, when the originalsignal is unavailable, the reader can estimate or predict the originalsignal based on properties of the watermarked signal. The original orpredicted version of the original signal can then be used to recover anestimate of the watermark message. One way to use the predicted versionto recover the watermark is to remove the predicted version beforereading the desired watermark. Similarly, the decoder can predict andremove un-wanted watermarks or watermark components before reading thedesired watermark in a signal having two or more watermarks.

2.1 Image Watermark Embedder

FIG. 2 is a block diagram illustrating an implementation of an exemplaryembedder in more detail. The embedding process begins with the message200. As noted above, the message is binary number suitable forconversion to a watermark signal. For additional security, the message,its carrier, and the mapping of the watermark to the host signal may beencrypted with an encryption key 202. In addition to the informationconveyed in the message, the embedder may also add control bit values(“signature bits”) to the message to assist in verifying the accuracy ofa read operation. These control bits, along with the bits representingthe message, are input to an error correction coding process 204designed to increase the likelihood that the message can be recoveredaccurately in the reader.

There are several alternative error correction coding schemes that maybe employed. Some examples include BCH, convolution, Reed Solomon andturbo codes. These forms of error correction coding are sometimes usedin communication applications where data is encoded in a carrier signalthat transfers the encoded data from one place to another. In thedigital watermarking application discussed here, the raw bit data isencoded in a fundamental carrier signal.

In addition to the error correction coding schemes mentioned above, theembedder and reader may also use a Cyclic Redundancy Check (CRC) tofacilitate detection of errors in the decoded message data.

The error correction coding function 204 produces a string of bits,termed raw bits 206, that are embedded into a watermark informationsignal. Using a carrier signal 208 and an assignment map 210, theillustrated embedder encodes the raw bits in a watermark informationsignal 212, 214. In some applications, the embedder may encode adifferent message in different locations of the signal. The carriersignal may be a noise image. For each raw bit, the assignment mapspecifies the corresponding image sample or samples that will bemodified to encode that bit.

The embedder depicted in FIG. 2 operates on blocks of image data(referred to as ‘tiles’) and replicates a watermark in each of theseblocks. As such, the carrier signal and assignment map both correspondto an image block of a pre-determined size, namely, the size of thetile. To encode each bit, the embedder applies the assignment map todetermine the corresponding image samples in the block to be modified toencode that bit. Using the map, it finds the corresponding image samplesin the carrier signal. For each bit, the embedder computes the value ofimage samples in the watermark information signal as a function of theraw bit value and the value(s) of the corresponding samples in thecarrier signal.

To illustrate the embedding process further, it is helpful to consideran example. First, consider the following background. Digitalwatermarking processes are sometimes described in terms of the transformdomain in which the watermark signal is defined. The watermark may bedefined in the spatial or temporal domain, or some other transformdomain such as a wavelet transform, Discrete Cosine Transform (DCT),Discrete Fourier Transform (DFT), Hadamard transform, Hartley transform,Karhunen-Loeve transform (KLT) domain, etc.

Consider an example where the watermark is defined in a transform domain(e.g., a frequency domain such as DCT, wavelet or DFT). The embeddersegments the image in the spatial domain into rectangular tiles andtransforms the image samples in each tile into the transform domain. Forexample in the DCT domain, the embedder segments the image into N by Nblocks and transforms each block into an N by N block of DCTcoefficients. In this example, the assignment map specifies thecorresponding sample location or locations in the frequency domain ofthe tile that correspond to a bit position in the raw bits. In thefrequency domain, the carrier signal looks like a noise pattern. Eachimage sample in the frequency domain of the carrier signal is usedtogether with a selected raw bit value to compute the value of the imagesample at the location in the watermark information signal.

Now consider an example where the watermark is defined in the spatialdomain. The embedder segments the image in the spatial domain intorectangular tiles of image samples (i.e. pixels). In this example, theassignment map specifies the corresponding sample location or locationsin the tile that correspond to each bit position in the raw bits. In thespatial domain, the carrier signal looks like a noise pattern extendingthroughout the tile. Each image sample in the spatial domain of thecarrier signal is used together with a selected raw bit value to computethe value of the image sample at the same location in the watermarkinformation signal.

With this background, the embedder proceeds to encode each raw bit inthe selected transform domain as follows. It uses the assignment map tolook up the position of the corresponding image sample (or samples) inthe carrier signal. The image sample value at that position in thecarrier controls the value of the corresponding position in thewatermark information signal. In particular, the carrier sample valueindicates whether to invert the corresponding watermark sample value.The raw bit value is either a one or zero. Disregarding for a moment theimpact of the carrier signal, the embedder adjusts the correspondingwatermark sample upward to represent a one, or downward to represent azero. Now, if the carrier signal indicates that the corresponding sampleshould be inverted, the embedder adjusts the watermark sample downwardto represent a one, and upward to represent a zero. In this manner, theembedder computes the value of the watermark samples for a raw bit usingthe assignment map to find the spatial location of those samples withinthe block.

From this example, a number of points can be made. First, the embeddermay perform a similar approach in any other transform domain. Second,for each raw bit, the corresponding watermark sample or samples are somefunction of the raw bit value and the carrier signal value. The specificmathematical relationship between the watermark sample, on one hand, andthe raw bit value and carrier signal, on the other, may vary with theimplementation. For example, the message may be convolved with thecarrier, multiplied with the carrier, added to the carrier, or appliedbased on another non-linear function. Third, the carrier signal mayremain constant for a particular application, or it may vary from onemessage to another. For example, a secret key may be used to generatethe carrier signal. For each raw bit, the assignment map may define apattern of watermark samples in the transform domain in which thewatermark is defined. An assignment map that maps a raw bit to a samplelocation or set of locations (i.e. a map to locations in a frequency orspatial domain) is just one special case of an assignment map for atransform domain. Fourth, the assignment map may remain constant, or itmay vary from one message to another. In addition, the carrier signaland map may vary depending on the nature of the underlying image. Insum, there many possible design choices within the implementationframework described above.

The embedder depicted in FIG. 2 combines another watermark component,shown as the detection watermark 216, with the watermark informationsignal to compute the final watermark signal. The detection watermark isspecifically chosen to assist in identifying the watermark and computingits orientation in a detection operation.

FIG. 3 is a spatial frequency plot illustrating one quadrant of adetection watermark. The points in the plot represent impulse functionsindicating signal content of the detection watermark signal. The patternof impulse functions for the illustrated quadrant is replicated in allfour quadrants. There are a number of properties of the detectionpattern that impact its effectiveness for a particular application. Theselection of these properties is highly dependent on the application.One property is the extent to which the pattern is symmetric about oneor more axes. For example, if the detection pattern is symmetrical aboutthe horizontal and vertical axes, it is referred to as being quadsymmetric. If it is further symmetrical about diagonal axes at an angleof 45 degrees, it is referred to as being octally symmetric (repeated ina symmetric pattern 8 times about the origin). Such symmetry aids inidentifying the watermark in an image, and aids in extracting therotation angle. However, in the case of an octally symmetric pattern,the detector includes an additional step of testing which of the fourquadrants the orientation angle falls into.

Another criterion is the position of the impulse functions and thefrequency range that they reside in. Preferably, the impulse functionsfall in a mid frequency range. If they are located in a low frequencyrange, they may be noticeable in the watermarked image. If they arelocated in the high frequency range, they are more difficult to recover.Also, they should be selected so that scaling, rotation, and othermanipulations of the watermarked signal do not push the impulsefunctions outside the range of the detector. Finally, the impulsefunctions should preferably not fall on the vertical or horizontal axes,and each impulse function should have a unique horizontal and verticallocation. While the example depicted in FIG. 3 shows that some of theimpulse functions fall on the same horizontal axis, it is trivial toalter the position of the impulse functions such that each has a uniquevertical or horizontal coordinate.

Returning to FIG. 2, the embedder makes a perceptual analysis 218 of theinput image 220 to identify portions of the image that can withstandmore watermark signal content without substantially impacting imagefidelity. Generally, the perceptual analysis employs a HVS model toidentify signal frequency bands and/or spatial areas to increase ordecrease watermark signal intensity to make the watermark imperceptibleto an ordinary observer. One type of model is to increase watermarkintensity in frequency bands and spatial areas where there is more imageactivity. In these areas, the sample values are changing more than otherareas and have more signal strength. The output of the perceptualanalysis is a perceptual mask 222. The mask may be implemented as anarray of functions, which selectively increase the signal strength ofthe watermark signal based on a HVS model analysis of the input image.The mask may selectively increase or decrease the signal strength of thewatermark signal in areas of greater signal activity.

The embedder combines (224) the watermark information, the detectionsignal and the perceptual mask to yield the watermark signal 226.Finally, it combines (228) the input image 220 and the watermark signal226 to create the watermarked image 230. In the frequency domainwatermark example above, the embedder combines the transform domaincoefficients in the watermark signal to the corresponding coefficientsin the input image to create a frequency domain representation of thewatermarked image. It then transforms the image into the spatial domain.As an alternative, the embedder may be designed to convert the watermarkinto the spatial domain, and then add it to the image.

In the spatial watermark example above, the embedder combines the imagesamples in the watermark signal to the corresponding samples in theinput image to create the watermarked image 230.

The embedder may employ an invertible or non-invertible, and linear ornon-linear function to combine the watermark signal and the input image(e.g., linear functions such as S*=S+gX; or S*=S(1+gX), convolution,quantization index modulation). The net effect is that some imagesamples in the input image are adjusted upward, while others areadjusted downward. The extent of the adjustment is greater in areas orsubbands of the image having greater signal activity.

2.2. Overview of a Detector and Reader

FIG. 4 is a flow diagram illustrating an overview of a watermarkdetection process. This process analyzes image data 400 to search for anorientation pattern of a watermark in an image suspected of containingthe watermark (the target image). First, the detector transforms theimage data to another domain 402, namely the spatial frequency domain,and then performs a series of correlation or other detection operations404. The correlation operations match the orientation pattern with thetarget image data to detect the presence of the watermark and itsorientation parameters 406 (e.g., translation, scale, rotation, anddifferential scale relative to its original orientation). Finally, itre-orients the image data based on one or more of the orientationparameters 408.

If the orientation of the watermark is recovered, the reader extractsthe watermark information signal from the image data (optionally byfirst re-orienting the data based on the orientation parameters). FIG. 5is flow diagram illustrating a process of extracting a message fromre-oriented image data 500. The reader scans the image samples (e.g.,pixels or transform domain coefficients) of the re-oriented image (502),and uses known attributes of the watermark signal to estimate watermarksignal values 504. Recall that in one example implementation describedabove, the embedder adjusted sample values (e.g., frequencycoefficients, color values, etc.) up or down to embed a watermarkinformation signal. The reader uses this attribute of the watermarkinformation signal to estimate its value from the target image. Prior tomaking these estimates, the reader may filter the image to removeportions of the image signal that may interfere with the estimatingprocess. For example, if the watermark signal is expected to reside inlow or medium frequency bands, then high frequencies may be filteredout.

In addition, the reader may predict the value of the originalun-watermarked image to enhance message recovery. One form of predictionuses temporal or spatial neighbors to estimate a sample value in theoriginal image. In the frequency domain, frequency coefficients of theoriginal signal can be predicted from neighboring frequency coefficientsin the same frequency subband. In video applications for example, afrequency coefficient in a frame can be predicted from spatiallyneighboring coefficients within the same frame, or temporallyneighboring coefficients in adjacent frames or fields. In the spatialdomain, intensity values of a pixel can be estimated from intensityvalues of neighboring pixels. Having predicted the value of a signal inthe original, un-watermarked image, the reader then estimates thewatermark signal by calculating an inverse of the watermarking functionused to combine the watermark signal with the original signal.

For such watermark signal estimates, the reader uses the assignment mapto find the corresponding raw bit position and image sample in thecarrier signal (506). The value of the raw bit is a function of thewatermark signal estimate, and the carrier signal at the correspondinglocation in the carrier. To estimate the raw bit value, the readersolves for its value based on the carrier signal and the watermarksignal estimate. As reflected generally in FIG. 5 (508), the result ofthis computation represents only one estimate to be analyzed along withother estimates impacting the value of the corresponding raw bit. Someestimates may indicate that the raw bit is likely to be a one, whileothers may indicate that it is a zero. After the reader completes itsscan, it compiles the estimates for each bit position in the raw bitstring, and makes a determination of the value of each bit at thatposition (510). Finally, it performs the inverse of the error correctioncoding scheme to construct the message (512). In some implementations,probablistic models may be employed to determine the likelihood that aparticular pattern of raw bits is just a random occurrence rather than awatermark.

2.2.1 Example Illustrating Detector Process

FIG. 6 is a diagram depicting an example of a watermark detectionprocess. The detector segments the target image into blocks (e.g., 600,602) and then performs a 2-dimensional fast fourier transform (2D FFT)on several blocks. This process yields 2D transforms of the magnitudesof the image contents of the blocks in the spatial frequency domain asdepicted in the plot 604 shown in FIG. 6.

Next, the detector process performs a log polar remapping of eachtransformed block. The detector may add some of the blocks together toincrease the watermark signal to noise ratio. The type of remapping inthis implementation is referred to as a Fourier Mellin transform. TheFourier Mellin transform is a geometric transform that warps the imagedata from a frequency domain to a log polar coordinate system. Asdepicted in the plot 606 shown in FIG. 6, this transform sweeps throughthe transformed image data along a line at angle θ, mapping the data toa log polar coordinate system shown in the next plot 608. The log polarcoordinate system has a rotation axis, representing the angle θ, and ascale axis. Inspecting the transformed data at this stage, one can seethe orientation pattern of the watermark begin to be distinguishablefrom the noise component (i.e., the image signal).

Next, the detector performs a correlation 610 between the transformedimage block and the transformed orientation pattern 612. At a highlevel, the correlation process slides the orientation pattern over thetransformed image (in a selected transform domain, such as a spatialfrequency domain) and measures the correlation at an array of discretepositions. Each such position has a corresponding scale and rotationparameter associated with it. Ideally, there is a position that clearlyhas the highest correlation relative to all of the others. In practice,there may be several candidates with a promising measure of correlation.As explained further below, these candidates may be subjected to one ormore additional correlation stages to select the one that provides thebest match.

There are a variety of ways to implement the correlation process. Anynumber of generalized matching filters may be implemented for thispurpose. One such filter performs an FFT on the target and theorientation pattern, and multiplies the resulting arrays together toyield a multiplied FFT. Finally, it performs an inverse FFT on themultiplied FFT to return the data into its original log-polar domain.The position or positions within this resulting array with the highestmagnitude represent the candidates with the highest correlation.

When there are several viable candidates, the detector can select a setof the top candidates and apply an additional correlation stage. Eachcandidate has a corresponding rotation and scale parameter. Thecorrelation stage rotates and scales the FFT of the orientation patternand performs a matching operation with the rotated and scaled pattern onthe FFT of the target image. The matching operation multiplies thevalues of the transformed pattern with sample values at correspondingpositions in the target image and accumulates the result to yield ameasure of the correlation. The detector repeats this process for eachof the candidates and picks the one with the highest measure ofcorrelation. As shown in FIG. 6, the rotation and scale parameters (614)of the selected candidate are then used to find additional parametersthat describe the orientation of the watermark in the target image.

The detector applies the scale and rotation to the target data block 616and then performs another correlation process between the orientationpattern 618 and the scaled and rotated data block 616. The correlationprocess 620 is a generalized matching filter operation. It provides ameasure of correlation for an array of positions that each has anassociated translation parameter (e.g., an x, y position). Again, thedetector may repeat the process of identifying promising candidates(i.e. those that reflect better correlation relative to others) andusing those in an additional search for a parameter or set oforientation parameters that provide a better measure of correlation.

At this point, the detector has recovered the following orientationparameters: rotation, scale and translation. For many applications,these parameters may be sufficient to enable accurate reading of thewatermark. In the read operation, the reader applies the orientationparameters to re-orient the target image and then proceeds to extractthe watermark signal.

In some applications, the watermarked image may be stretched more in onespatial dimension than another. This type of distortion is sometimesreferred to as differential scale or shear. Consider that the originalimage blocks are square. As a result of differential scale, each squaremay be warped into a parallelogram with unequal sides. Differentialscale parameters define the nature and extent of this stretching.

There are several alternative ways to recover the differential scaleparameters. One general class of techniques is to use the knownparameters (e.g., the computed scale, rotation, and translation) as astarting point to find the differential scale parameters. Assuming theknown parameters to be valid, this approach warps either the orientationpattern or the target image with selected amounts of differential scaleand picks the differential scale parameters that yield the bestcorrelation.

Another approach to determination of differential scale is set forth inapplication Ser. No. 09/452,022 (filed Nov. 30, 1999, and entitledMethod and System for Determining Image Transformation).

2.2.2 Example Illustrating Reader Process

FIG. 7 is a diagram illustrating a re-oriented image 700 superimposedonto the original watermarked image 702. The difference in orientationand scale shows how the image was transformed and edited after theembedding process. The original watermarked image is sub-divided intotiles (e.g., pixel blocks 704, 706, etc.). When superimposed on thecoordinate system of the original image 702 shown in FIG. 7, the targetimage blocks typically do not match the orientation of the originalblocks.

The reader scans samples of the re-oriented image data, estimating thewatermark information signal. It estimates the watermark informationsignal, in part, by predicting original sample values of theun-watermarked image. The reader then uses an inverted form of thewatermarking function to estimate the watermark information signal fromthe watermarked signal and the predicted signal. This invertedwatermarking function expresses the estimate of the watermark signal asa function of the predicted signal and the watermarked signal. Having anestimate of the watermark signal, it then uses the known relationshipamong the carrier signal, the watermark signal, and the raw bit tocompute an estimate of the raw bit. Recall that samples in the watermarkinformation signal are a function of the carrier signal and the raw bitvalue. Thus, the reader may invert this function to solve for anestimate of the raw bit value.

Recall that the embedder implementation discussed in connection withFIG. 2 redundantly encodes the watermark information signal in blocks ofthe input signal. Each raw bit may map to several samples within ablock. In addition, the embedder repeats a mapping process for each ofthe blocks. As such, the reader generates several estimates of the rawbit value as it scans the watermarked image.

The information encoded in the raw bit string can be used to increasethe accuracy of read operations. For instance, some of the raw bits actas signature bits that perform a validity checking function. Unlikeunknown message bits, the reader knows the expected values of thesesignature bits. The reader can assess the validity of a read operationbased on the extent to which the extracted signature bit values matchthe expected signature bit values. The estimates for a given raw bitvalue can then be given a higher weight depending on whether they arederived from a tile with a greater measure of validity.

3.0 Embedder Implementation:

The following sections describe an implementation of the digital imagewatermark embedder depicted in FIG. 8. The embedder inserts twowatermark components into the host image: a message component and adetection component (called the orientation pattern). The messagecomponent is defined in a spatial domain or other transform domain,while the orientation pattern is defined in a frequency domain. Asexplained later, the message component serves a dual function ofconveying a message and helping to identify the watermark location inthe image.

The embedder inserts the watermark message and orientation pattern inblocks of a selected color plane or planes (e.g., luminance orchrominance plane) of the host image. The message payload varies fromone application to another, and can range from a single bit to thenumber of image samples in the domain in which it is embedded. Theblocks may be blocks of samples in a spatial domain or some othertransform domain.

3.1 Encoding the Message

The embedder converts binary message bits into a series of binary rawbits that it hides in the host image. As part of this process, a messageencoder 800 appends certain known bits to the message bits 802. Itperforms an error detection process (e.g., parity, Cyclic RedundancyCheck (CRC), etc.) to generate error detection bits and adds the errordetection bits to the message. An error correction coding operation thengenerates raw bits from the combined known and message bit string.

For the error correction operation, the embedder may employ any of avariety of error correction codes such as Reed Solomon, BCH, convolutionor turbo codes. The encoder may perform an M-ary modulation process onthe message bits that maps groups of message bits to a message signalbased on an M-ary symbol alphabet.

In one application of the embedder, the component of the messagerepresenting the known bits is encoded more redundantly than the othermessage bits. This is an example of a shorter message component havinggreater signal strength than a longer, weaker message component. Theembedder gives priority to the known bits in this scheme because thereader uses them to verify that it has found the watermark in apotentially corrupted image, rather than a signal masquerading as thewatermark.

3.2 Spread Spectrum Modulation

The embedder uses spread spectrum modulation as part of the process ofcreating a watermark signal from the raw bits. A spread spectrummodulator 804 spreads each raw bit into a number of “chips.” Theembedder generates a pseudo random number that acts as the carriersignal of the message. To spread each raw bit, the modulator performs anexclusive OR (XOR) operation between the raw bit and each bit of apseudo random binary number of a pre-determined length. The length ofthe pseudo random number depends, in part, on the size of the messageand the image. Preferably, the pseudo random number should containroughly the same number of zeros and ones, so that the net effect of theraw bit on the host image block is zero. If a bit value in the pseudorandom number is a one, the value of the raw bit is inverted.Conversely, if the bit value is a zero, then the value of the raw bitremains the same.

The length of the pseudorandom number may vary from one message bit orsymbol to another. By varying the length of the number, some messagebits can be spread more than others.

3.3 Scattering the Watermark Message

The embedder scatters each of the chips corresponding to a raw bitthroughout an image block. An assignment map 806 assigns locations inthe block to the chips of each raw bit. Each raw bit is spread overseveral chips. As noted above, an image block may represent a block oftransform domain coefficients or samples in a spatial domain. Theassignment map may be used to encode some message bits or symbols (e.g.,groups of bits) more redundantly than others by mapping selected bits tomore locations in the host signal than other message bits. In addition,it may be used to map different messages, or different components of thesame message, to different locations in the host signal.

FIG. 9 depicts an example of the assignment map. Each of the blocks inFIG. 9 correspond to an image block and depict a pattern of chipscorresponding to a single raw bit. FIG. 9 depicts a total of 32 exampleblocks. The pattern within a block is represented as white dots on ablack background. Each of the patterns is mutually exclusive such thateach raw bit maps to a pattern of unique locations relative to thepatterns of every other raw bit. Though not a requirement, the combinedpatterns, when overlapped, cover every location within the image block.

3.4 Gain Control and Perceptual Analysis

To insert the information carried in a chip to the host image, theembedder alters the corresponding sample value in the host image. Inparticular, for a chip having a value of one, it adds to thecorresponding sample value, and for a chip having a value of zero, itsubtracts from the corresponding sample value. A gain controller in theembedder adjusts the extent to which each chip adds or subtracts fromthe corresponding sample value.

The gain controller takes into account the orientation pattern whendetermining the gain. It applies a different gain to the orientationpattern than to the message component of the watermark. After applyingthe gain, the embedder combines the orientation pattern and messagecomponents together to form the composite watermark signal, and combinesthe composite watermark with the image block. One way to combine thesesignal components is to add them, but other linear or non-linearfunctions may be used as well.

The orientation pattern is comprised of a pattern of quad symmetricimpulse functions in the spatial frequency domain. In the spatialdomain, these impulse functions look like cosine waves. An example ofthe orientation pattern is depicted in FIGS. 10 and 11. FIG. 10 showsthe impulse functions as points in the spatial frequency domain, whileFIG. 11 shows the orientation pattern in the spatial domain. Beforeadding the orientation pattern component to the message component, theembedder may transform the watermark components to a common domain. Forexample, if the message component is in a spatial domain and theorientation component is in a frequency domain, the embedder transformsthe orientation component to a common spatial domain before combiningthem together.

FIG. 8 depicts the gain controller used in the embedder. Note that thegain controller operates on the blocks of image samples 808, the messagewatermark signal, and a global gain input 810, which may be specified bythe user. A perceptual analyzer component 812 of the gain controllerperforms a perceptual analysis on the block to identify samples that cantolerate a stronger watermark signal without substantially impactingvisibility. In places where the naked eye is less likely to notice thewatermark, the perceptual analyzer increases the strength of thewatermark. Conversely, it decreases the watermark strength where the eyeis more likely to notice the watermark.

The perceptual analyzer shown in FIG. 8 performs a series of filteringoperations on the image block to compute an array of gain values. Thereare a variety of filters suitable for this task. These filters includean edge detector filter that identifies edges of objects in the image, anon-linear filter to map gain values into a desired range, and averagingor median filters to smooth the gain values. Each of these filters maybe implemented as a series of one-dimensional filters (one operating onrows and the other on columns) or two-dimensional filters. The size ofthe filters (i.e. the number of samples processed to compute a value fora given location) may vary (e.g., 3 by 3, 5 by 5, etc.). The shape ofthe filters may vary as well (e.g., square, cross-shaped, etc.). Theperceptual analyzer process produces a detailed gain multiplier. Themultiplier is a vector with elements corresponding to samples in ablock.

Another component 818 of the gain controller computes an asymmetric gainbased on the output of the image sample values and message watermarksignal. This component analyzes the samples of the block to determinewhether they are consistent with the message signal. The embedderreduces the gain for samples whose values relative to neighboring valuesare consistent with the message signal.

The embedder applies the asymmetric gain to increase the chances of anaccurate read in the watermark reader. To understand the effect of theasymmetric gain, it is helpful to explain the operation of the reader.The reader extracts the watermark message signal from the watermarkedsignal using a predicted version of the original signal. It estimatesthe watermark message signal value based on values of the predictedsignal and the watermarked signal at locations of the watermarked signalsuspected of containing a watermark signal. There are several ways topredict the original signal. One way is to compute a local average ofsamples around the sample of interest. The average may be computed bytaking the average of vertically adjacent samples, horizontally adjacentsamples, an average of samples in a cross-shaped filter (both verticaland horizontal neighbors, an average of samples in a square-shapedfilter, etc. The estimate may be computed one time based on a singlepredicted value from one of these averaging computations. Alternatively,several estimates may be computed based on two or more of theseaveraging computations (e.g., one estimate for vertically adjacentsamples and another for horizontally adjacent samples). In the lattercase, the reader may keep estimates if they satisfy a similarity metric.In other words, the estimates are deemed valid if they within apredetermined value or have the same polarity.

Knowing this behavior of the reader, the embedder computes theasymmetric gain as follows. For samples that have values relative totheir neighbors that are consistent with the watermark signal, theembedder reduces the asymmetric gain. Conversely, for samples that areinconsistent with the watermark signal, the embedder increases theasymmetric gain. For example, if the chip value is a one, then thesample is consistent with the watermark signal if its value is greaterthan its neighbors. Alternatively, if the chip value is a zero, then thesample is consistent with the watermark signal if its value is less thanits neighbors.

Another component 820 of the gain controller computes a differentialgain, which represents an adjustment in the message vs. orientationpattern gains. As the global gain increases, the embedder emphasizes themessage gain over the orientation pattern gain by adjusting the globalgain by an adjustment factor. The inputs to this process 820 include theglobal gain 810 and a message differential gain 822. When the globalgain is below a lower threshold, the adjustment factor is one. When theglobal gain is above an upper threshold, the adjustment factor is set toan upper limit greater than one. For global gains falling within the twothresholds, the adjustment factor increases linearly between one and theupper limit. The message differential gain is the product of theadjustment factor and the global gain.

At this point, there are four sources of gain: the detailed gain, theglobal gain, the asymmetric gain, and the message dependent gain. Theembedder applies the first two gain quantities to both the message andorientation watermark signals. It only applies the latter two to themessage watermark signal. FIG. 8 depicts how the embedder applies thegain to the two watermark components. First, it multiplies the detailedgain with the global gain to compute the orientation pattern gain. Itthen multiplies the orientation pattern gain with the adjusted messagedifferential gain and asymmetric gain to form the composite messagegain.

Finally, the embedder forms the composite watermark signal. Itmultiplies the composite message gain with the message signal, andmultiplies the orientation pattern gain with the orientation patternsignal. It then combines the result in a common transform domain to getthe composite watermark. The embedder applies a watermarking function tocombine the composite watermark to the block to create a watermarkedimage block. The message and orientation components of the watermark maycombined by mapping the message bits to samples of the orientationsignal, and modulating the samples of the orientation signal to encodethe message.

The embedder computes the watermark message signal by converting theoutput of the assignment map 806 to delta values, indicating the extentto which the watermark signal changes the host signal. As noted above, achip value of one corresponds to an upward adjustment of thecorresponding sample, while a chip value of zero corresponds to adownward adjustment. The embedder specifies the specific amount ofadjustment by assigning a delta value to each of the watermark messagesamples (830).

4.0 Detector Implementation

FIG. 12 illustrates an overview of a watermark detector that detects thepresence of a detection watermark in a host image and its orientation.Using the orientation pattern and the known bits inserted in thewatermark message, the detector determines whether a potentiallycorrupted image contains a watermark, and if so, its orientation in theimage.

Recall that the composite watermark is replicated in blocks of theoriginal image. After an embedder places the watermark in the originaldigital image, the watermarked image is likely to undergo severaltransformations, either from routine processing or from intentionaltampering. Some of these transformations include: compression,decompression, color space conversion, digital to analog conversion,printing, scanning, analog to digital conversion, scaling, rotation,inversion, flipping differential scale, and lens distortion. In additionto these transformations, various noise sources can corrupt thewatermark signal, such as fixed pattern noise, thermal noise, etc.

When building a detector implementation for a particular application,the developer may implement counter-measures to mitigate the impact ofthe types of transformations, distortions and noise expected for thatapplication. Some applications may require more counter-measures thanothers. The detector described below is designed to recover a watermarkfrom a watermarked image after the image has been printed, and scanned.The following sections describe the counter-measures to mitigate theimpact of various forms of corruption. The developer can select fromamong these counter-measures when implementing a detector for aparticular application.

For some applications, the detector will operate in a system thatprovides multiple image frames of a watermarked object. One typicalexample of such a system is a computer equipped with a digital camera.In such a configuration, the digital camera can capture a temporalsequence of images as the user or some device presents the watermarkedimage to the camera.

As shown in FIG. 12, the principal components of the detector are: 1)pre-processor 900; 2) rotation and scale estimator 902; 3) orientationparameter refiner 904; 4) translation estimator 906; 5) translationrefiner 908; and reader 910.

The preprocessor 900 takes one or more frames of image data 912 andproduces a set of image blocks 914 prepared for further analysis. Therotation-scale estimator 902 computes rotation-scale vectors 916 thatestimate the orientation of the orientation signal in the image blocks.The parameter refiner 904 collects additional evidence of theorientation signal and further refines the rotation scale vectorcandidates by estimating differential scale parameters. The result ofthis refining stage is a set of 4D vectors candidates 918 (rotation,scale, and two differential scale parameters). The translation estimator906 uses the 4D vector candidates to re-orient image blocks withpromising evidence of the orientation signal. It then finds estimates oftranslation parameters 920. The translation refiner 908 invokes thereader 910 to assess the merits of an orientation vector. When invokedby the detector, the reader uses the orientation vector to approximatethe original orientation of the host image and then extracts values forthe known bits in the watermark message. The detector uses thisinformation to assess the merits of and refine orientation vectorcandidates.

By comparing the extracted values of the known bits with the expectedvalues, the reader provides a figure of merit for an orientation vectorcandidate. The translation refiner then picks a 6D vector, includingrotation, scale, differential scale and translation, that appears likelyproduce a valid read of the watermark message 922. The followingsections describe implementations of these components in more detail.

4.1 Detector Pre-processing

FIG. 13 is a flow diagram illustrating preprocessing operations in thedetector shown in FIG. 12. The detector performs a series ofpre-processing operations on the native image 930 to prepare the imagedata for further analysis. It begins by filling memory with one or moreframes of native image data (932), and selecting sets of pixel blocks934 from the native image data for further analysis (936). While thedetector can detect a watermark using a single image frame, it also hassupport for detecting the watermark using additional image frames. Asexplained below, the use of multiple frames has the potential forincreasing the chances of an accurate detection and read.

In applications where a camera captures an input image of a watermarkedobject, the detector may be optimized to address problems resulting frommovement of the object. Typical PC cameras, for example, are capable ofcapturing images at a rate of at least 10 frames a second. A frustrateduser might attempt to move the object in an attempt to improvedetection. Rather than improving the chances of detection, the movementof the object changes the orientation of the watermark from one frame tothe next, potentially making the watermark more difficult to detect. Oneway to address this problem is to buffer one or more frames, and thenscreen the frame or frames to determine if they are likely to contain avalid watermark signal. If such screening indicates that a frame is notlikely to contain a valid signal, the detector can discard it andproceed to the next frame in the buffer, or buffer a new frame. Anotherenhancement is to isolate portions of a frame that are most likely tohave a valid watermark signal, and then perform more detailed detectionof the isolated portions.

After loading the image into the memory, the detector selects imageblocks 934 for further analysis. It is not necessary to load or examineeach block in a frame because it is possible to extract the watermarkusing only a portion of an image. The detector looks at only a subset ofthe samples in an image, and preferably analyzes samples that are morelikely to have a recoverable watermark signal.

The detector identifies portions of the image that are likely to havethe highest watermark signal to noise ratio. It then attempts to detectthe watermark signal in the identified portions. In the context ofwatermark detection, the host image is considered to be a source ofnoise along with conventional noise sources. While it is typically notpractical to compute the signal to noise ratio, the detector canevaluate attributes of the signal that are likely to evince a promisingwatermark signal to noise ratio. These properties include the signalactivity (as measured by sample variance, for example), and a measure ofthe edges (abrupt changes in image sample values) in an image block.Preferably, the signal activity of a candidate block should fall withinan acceptable range, and the block should not have a high concentrationof strong edges. One way to quantify the edges in the block is to use anedge detection filter (e.g., a LaPlacian, Sobel, etc.).

In one implementation, the detector divides the input image into blocks,and analyzes each block based on pre-determined metrics. It then ranksthe blocks according to these metrics. The detector then operates on theblocks in the order of the ranking. The metrics include sample variancein a candidate block and a measure of the edges in the block. Thedetector combines these metrics for each candidate block to compute arank representing the probability that it contains a recoverablewatermark signal.

In another implementation, the detector selects a pattern of blocks andevaluates each one to try to make the most accurate read from theavailable data. In either implementation, the block pattern and size mayvary. This particular implementation selects a pattern of overlappingblocks (e.g., a row of horizontally aligned, overlapping blocks). Oneoptimization of this approach is to adaptively select a block patternthat increases the signal to noise ratio of the watermark signal. Whileshown as one of the initial operations in the preparation, the selectionof blocks can be postponed until later in the pre-processing stage.

Next, the detector performs a color space conversion on native imagedata to compute an array of image samples in a selected color space foreach block (936). In the following description, the color space isluminance, but the watermark may be encoded in one or more differentcolor spaces. The objective is to get a block of image samples withlowest noise practical for the application. While the implementationcurrently performs a row by row conversion of the native image data into8 bit integer luminance values, it may be preferable to convert tofloating-point values for some applications. One optimization is toselect a luminance converter that is adapted for the sensor used tocapture the digital input image. For example, one might experimentallyderive the lowest noise luminance conversion for commercially availablesensors, e.g., CCD cameras or scanners, CMOS cameras, etc. Then, thedetector could be programmed to select either a default luminanceconverter, or one tuned to a specific type of sensor.

At one or more stages of the detector, it may be useful to performoperations to mitigate the impact of noise and distortion. In thepre-processing phase, for example, it may be useful to evaluate fixedpattern noise and mitigate its effect (938). The detector may look forfixed pattern noise in the native input data or the luminance data, andthen mitigate it.

One way to mitigate certain types of noise is to combine data fromdifferent blocks in the same frame, or corresponding blocks in differentframes 940. This process helps augment the watermark signal present inthe blocks, while reducing the noise common to the blocks. For example,merely adding blocks together may mitigate the effects of common noise.

In addition to common noise, other forms of noise may appear in each ofthe blocks such as noise introduced in the printing or scanningprocesses. Depending on the nature of the application, it may beadvantageous to perform common noise recognition and removal at thisstage 942. The developer may select a filter or series of filters totarget certain types of noise that appear during experimentation withimages. Certain types of median filters may be effective in mitigatingthe impact of spectral peaks (e.g., speckles) introduced in printing orscanning operations.

In addition to introducing noise, the printing and image captureprocesses may transform the color or orientation of the original,watermarked image. As described above, the embedder typically operateson a digital image in a particular color space and at a desiredresolution. The watermark embedders normally operate on digital imagesrepresented in an RGB or CYMK color space at a desired resolution (e.g.,100 dpi or 300 dpi, the resolution at which the image is printed). Theimages are then printed on paper with a screen printing process thatuses the CYMK subtractive color space at a line per inch (LPI) rangingfrom 65-200. 133 lines/in is typical for quality magazines and 73lines/in is typical for newspapers. In order to produce a quality imageand avoid pixelization, the rule of thumb is to use digital images witha resolution that is at least twice the press resolution. This is due tothe half tone printing for color production. Also, different presses usescreens with different patterns and line orientations and have differentprecision for color registration.

One way to counteract the transforms introduced through the printingprocess is to develop a model that characterizes these transforms andoptimize watermark embedding and detecting based on thischaracterization. Such a model may be developed by passing watermarkedand unwatermarked images through the printing process and observing thechanges that occur to these images. The resulting model characterizesthe changes introduced due to the printing process. The model mayrepresent a transfer function that approximates the transforms due tothe printing process. The detector then implements a pre-processingstage that reverses or at least mitigates the effect of the printingprocess on watermarked images. The detector may implement apre-processing stage that performs the inverse of the transfer functionfor the printing process.

A related challenge is the variety in paper attributes used in differentprinting processes. Papers of various qualities, thickness andstiffness, absorb ink in various ways. Some papers absorb ink evenly,while others absorb ink at rates that vary with the changes in thepaper's texture and thickness. These variations may degrade the embeddedwatermark signal when a digitally watermarked image is printed. Thewatermark process can counteract these effects by classifying andcharacterizing paper so that the embedder and reader can compensate forthis printing-related degradation.

Variations in image capture processes also pose a challenge. In someapplications, it is necessary to address problems introduced due tointerlaced image data. Some video camera produce interlaced fieldsrepresenting the odd or even scan lines of a frame. Problems arise whenthe interlaced image data consists of fields from two consecutiveframes. To construct an entire frame, the preprocessor may combine thefields from consecutive frames while dealing with the distortion due tomotion that occurs from one frame to the next. For example, it may benecessary to shift one field before interleaving it with another fieldto counteract inter-frame motion. A de-blurring function may be used tomitigate the blurring effect due to the motion between frames.

Another problem associated with cameras in some applications is blurringdue to the lack of focus. The preprocessor can mitigate this effect byestimating parameters of a blurring function and applying a de-blurringfunction to the input image.

Yet another problem associated with cameras is that they tend to havecolor sensors that utilize different color pattern implementations. Assuch, a sensor may produce colors slightly different than thoserepresented in the object being captured. Most CCD and CMOS cameras usean array of sensors to produce colored images. The sensors in the arrayare arranged in clusters of sensitive to three primary colors red,green, and blue according to a specific pattern. Sensors designated fora particular color are dyed with that color to increase theirsensitivity to the designated color. Many camera manufacturers use aBayer color pattern GR/BG. While this pattern produces good imagequality, it causes color mis-registration that degrades the watermarksignal. Moreover, the color space converter, which maps the signal fromthe sensors to another color space such as YUV or RGB, may vary from onemanufacturer to another. One way to counteract the mis-registration ofthe camera's color pattern is to account for the distortion due to thepattern in a color transformation process, implemented either within thecamera itself, or as a pre-processing function in the detector.

Another challenge in counteracting the effects of the image captureprocess is dealing with the different types of distortion introducedfrom various image capture devices. For example, cameras have differentsensitivities to light. In addition, their lenses have differentspherical distortion, and noise characteristics. Some scanners have poorcolor reproduction or introduce distortion in the image aspect ratio.Some scanners introduce aliasing and employ interpolation to increaseresolution. The detector can counteract these effects in thepre-processor by using an appropriate inverse transfer function. Anoff-line process first characterizes the distortion of several differentimage capture devices (e.g., by passing test images through the scannerand deriving a transfer function modeling the scanner distortion). Somedetectors may be equipped with a library of such inverse transferfunctions from which they select one that corresponds to the particularimage capture device

Yet another challenge in applications where the image is printed onpaper and later scanned is that the paper deteriorates over time anddegrades the watermark. Also, varying lighting conditions make thewatermark difficult to detect. Thus, the watermark may be selected so asto be more impervious to expected deterioration, and recoverable over awider range of lighting conditions.

At the close of the pre-processing stage, the detector has selected aset of blocks for further processing. It then proceeds to gatherevidence of the orientation signal in these blocks, and estimate theorientation parameters of promising orientation signal candidates. Sincethe image may have suffered various forms of corruption, the detectormay identify several parts of the image that appear to have attributessimilar to the orientation signal. As such, the detector may have toresolve potentially conflicting and ambiguous evidence of theorientation signal. To address this challenge, the detector estimatesorientation parameters, and then refines theses estimates to extract theorientation parameters that are more likely to evince a valid signalthan other parameter candidates.

4.2 Estimating Initial Orientation Parameters

FIG. 14 is a flow diagram illustrating a process for estimatingrotation-scale vectors. The detector loops over each image block (950),calculating rotation-scale vectors with the best detection values ineach block. First, the detector filters the block in a manner that tendsto amplify the orientation signal while suppressing noise, includingnoise from the host image itself (952). Implemented as a multi-axisLaPlacian filter, the filter highlights edges (e.g., high frequencycomponents of the image) and then suppresses them. The term,“multi-axis,” means that the filter includes a series of stages thateach operates on particular axis. First, the filter operates on the rowsof luminance samples, then operates on the columns, and adds theresults. The filter may be applied along other axes as well. Each passof the filter produces values at discrete levels. The final result is anarray of samples, each having one of five values: {−2, −1, 0, 1, 2}.

Next, the detector performs a windowing operation on the block data toprepare it for an FFT transform (954). This windowing operation providessignal continuity at the block edges. The detector then performs an FFT(956) on the block, and retains only the magnitude component (958).

In an alternative implementation, the detector may use the phase signalproduced by the FFT to estimate the translation parameter of theorientation signal. For example, the detector could use the rotation andscale parameters extracted in the process described below, and thencompute the phase that provided the highest measure of correlation withthe orientation signal using the phase component of the FFT process.

After computing the FFT, the detector applies a Fourier magnitude filter(960) on the magnitude components. The filter in the implementationslides over each sample in the Fourier magnitude array and filters thesample's eight neighbors in a square neighborhood centered at thesample. The filter boosts values representing a sharp peak with a rapidfall-off, and suppresses the fall-off portion. It also performs athreshold operation to clip peaks to an upper threshold.

Next, the detector performs a log-polar re-sample (962) of the filteredFourier magnitude array to produce a log-polar array 964. This type ofoperation is sometimes referred to as a Fourier Mellin transform. Thedetector, or some off-line pre-processor, performs a similar operationon the orientation signal to map it to the log-polar coordinate system.Using matching filters, the detector implementation searches for aorientation signal in a specified window of the log-polar coordinatesystem. For example, consider that the log-polar coordinate system is atwo dimensional space with the scale being the vertical axis and theangle being the horizontal axis. The window ranges from 0 to 90 degreeson the horizontal axis and from approximately 50 to 2400 dpi on thevertical axis. Note that the orientation pattern should be selected sothat routine scaling does not push the orientation pattern out of thiswindow. The orientation pattern can be designed to mitigate thisproblem, as noted above, and as explained in co-pending patentapplication No. 60/136,572, filed May 28, 1999, by Ammon Gustafson,entitled Watermarking System With Improved Technique for DetectingScaling and Rotation, filed May 28, 1999.

The detector proceeds to correlate the orientation and the target signalin the log polar coordinate system. As shown in FIG. 14, the detectoruses a generalized matched filter GMF (966). The GMF performs an FFT onthe orientation and target signal, multiplies the resulting Fourierdomain entities, and performs an inverse FFT. This process yields arectangular array of values in log-polar coordinates, each representinga measure of correlation and having a corresponding rotation angle andscale vector. As an optimization, the detector may also perform the samecorrelation operations for distorted versions (968, 970, 972) of theorientation signal to see if any of the distorted orientation patternsresults in a higher measure of correlation. For example, the detectormay repeat the correlation operation with some pre-determined amount ofhorizontal and vertical differential distortion (970, 972). The resultof this correlation process is an array of correlation values 974specifying the amount of correlation that each correspondingrotation-scale vector provides.

The detector processes this array to find the top M peaks and theirlocation in the log-polar space 976. To extract the location moreaccurately, the detector uses interpolation to provide the inter-samplelocation of each of the top peaks 978. The interpolator computes the 2Dmedian of the samples around a peak and provides the location of thepeak center to an accuracy of 0.1 sample.

The detector proceeds to rank the top rotation-scale vectors based onyet another correlation process 980. In particular, the detectorperforms a correlation between a Fourier magnitude representation foreach rotation-scale vector candidate and a Fourier magnitudespecification of the orientation signal 982. Each Fourier magnituderepresentation is scaled and rotated by an amount reflected by thecorresponding rotation-scale vector. This correlation operation sums apoint-wise multiplication of the orientation pattern impulse functionsin the frequency domain with the Fourier magnitude values of the imageat corresponding frequencies to compute a measure of correlation foreach peak 984. The detector then sorts correlation values for the peaks(986).

Finally, the detector computes a detection value for each peak (988). Itcomputes the detection value by quantizing the correlation values.Specifically, it computes a ratio of the peak's correlation value andthe correlation value of the next largest peak. Alternatively, thedetector may compute the ratio of the peak's correlation value and a sumor average of the correlation values of the next n highest peaks, wheren is some predetermined number. Then, the detector maps this ratio to adetection value based on a statistical analysis of unmarked images.

The statistical analysis plots a distribution of peak ratio values foundin unmarked images. The ratio values are mapped to a detection valuebased on the probability that the value came from an unmarked image. Forexample, 90% of the ratio values in unmarked images fall below a firstthreshold T1, and thus, the detection value mapping for a ratio of T1 isset to 1. Similarly, 99% of the ratio values in unmarked images fallbelow T2, and therefore, the detection value is set to 2. 99.9% of theratio values in unmarked images fall below T3, and the correspondingdetection value is set to 3. The threshold values, T1, T2 and T3, may bedetermined by performing a statistical analysis of several images. Themapping of ratios to detection values based on the statisticaldistribution may be implemented in a look up table.

The statistical analysis may also include a maximum likelihood analysis.In such an analysis, an off-line detector generates detection valuestatistics for both marked and unmarked images. Based on the probabilitydistributions of marked and unmarked images, it determines thelikelihood that a given detection value for an input image originatesfrom a marked and unmarked image.

At the end of these correlation stages, the detector has computed aranked set of rotation-scale vectors 990, each with a quantized measureof correlation associated with it. At this point, the detector couldsimply choose the rotation and scale vectors with the highest rank andproceed to compute other orientation parameters, such as differentialscale and translation. Instead, the detector gathers more evidence torefine the rotation-scale vector estimates. FIG. 15 is a flow diagramillustrating a process for refining the orientation parameters usingevidence of the orientation signal collected from blocks in the currentframe.

Continuing in the current frame, the detector proceeds to compare therotation and scale parameters from different blocks (e.g., block 0,block 1, block 2; 1000, 1002, and 1004 in FIG. 15). In a processreferred to as interblock coincidence matching 1006, it looks forsimilarities between rotation-scale parameters that yielded the highestcorrelation in different blocks. To quantify this similarity, itcomputes the geometric distance between each peak in one block withevery other peak in the other blocks. It then computes the probabilitythat peaks will fall within this calculated distance. There are avariety of ways to calculate the probability. In one implementation, thedetector computes the geometric distance between two peaks, computes thecircular area encompassing the two peaks (π(geometric distance)²), andcomputes the ratio of this area to the total area of the block. Finally,it quantizes this probability measure for each pair of peaks (1008) bycomputing the log(base 10) of the ratio of the total area over the areaencompassing the two peaks. At this point, the detector has calculatedtwo detection values: quantized peak value, and the quantized distancemetric.

The detector now forms multi-block grouping of rotation-scale vectorsand computes a combined detection value for each grouping (1010). Thedetector groups vectors based on their relative geometric proximitywithin their respective blocks. It then computes the combined detectionvalue by combining the detection values of the vectors in the group(1012). One way to compute a combined detection value is to add thedetection values or add a weighted combination of them.

Having calculated the combined detection values, the detector sorts eachgrouping by its combined detection value (1014). This process produces aset of the top groupings of unrefined rotation-scale candidates, rankedby detection value 1016. Next, the detector weeds out rotation-scalevectors that are not promising by excluding those groupings whosecombined detection values are below a threshold (the “refine threshold”1018). The detector then refines each individual rotation-scale vectorcandidate within the remaining groupings.

The detector refines a rotation-scale vector by adjusting the vector andchecking to see whether the adjustment results in a better correlation.As noted above, the detector may simply pick the best rotation-scalevector based on the evidence collected thus far, and refine only thatvector. An alternative approach is to refine each of the toprotation-scale vector candidates, and continue to gather evidence foreach candidate. In this approach, the detector loops over each vectorcandidate (1020), refining each one.

One approach of refining the orientation vector is as follows:

-   -   fix the orientation signal impulse functions (“points”) within a        valid boundary (1022);    -   pre-refine the rotation-scale vector (1024);    -   find the major axis and re-fix the orientation points (1026);        and    -   refine each vector with the addition of a differential scale        component (1028).

In this approach, the detector pre-refines a rotation-scale vector byincrementally adjusting one of the parameters (scale, rotation angle),adjusting the orientation points, and then summing a point-wisemultiplication of the orientation pattern and the image block in theFourier magnitude domain. The refiner compares the resulting measure ofcorrelation with previous measures and continues to adjust one of theparameters so long as the correlation increases. After refining thescale and rotation angle parameters, the refiner finds the major axis,and re-fixes the orientation points. It then repeats the refiningprocess with the introduction of differential scale parameters. At theend of this process, the refiner has converted each scale-rotationcandidate to a refined 4D vector, including rotation, scale, and twodifferential scale parameters.

At this stage, the detector can pick a 4D vector or set of 4D vector andproceed to calculate the final remaining parameter, translation.Alternatively, the detector can collect additional evidence about themerits of each 4D vector.

One way to collect additional evidence about each 4D vector is tore-compute the detection value of each orientation vector candidate(1030). For example, the detector may quantize the correlation valueassociated with each 4D vector as described above for the rotation-scalevector peaks (see item 988, FIG. 14 and accompanying text). Another wayto collect additional evidence is to repeat the coincidence matchingprocess for the 4D vectors. For this coincidence matching process, thedetector computes spatial domain vectors for each candidate (1032),determines the distance metric between candidates from different blocks,and then groups candidates from different blocks based on the distancemetrics (1034). The detector then re-sorts the groups according to theircombined detection values (1036) to produce a set of the top P groupings1038 for the frame.

FIG. 16 is a flow diagram illustrating a method for aggregating evidenceof the orientation signal from multiple frames. In applications withmultiple frames, the detector collects the same information fororientation vectors of the selected blocks in each frame (namely, thetop P groupings of orientation vector candidates, e.g., 1050, 1052 and1054). The detector then repeats coincidence matching betweenorientation vectors of different frames (1056). In particular, in thisinter-frame mode, the detector quantizes the distance metrics computedbetween orientation vectors from blocks in different frames (1058). Itthen finds inter-frame groupings of orientation vectors (super-groups)using the same approach described above (1060), except that theorientation vectors are derived from blocks in different frames. Afterorganizing orientation vectors into super-groups, the detector computesa combined detection value for each super-group (1062) and sorts thesuper-groups by this detection value (1064). The detector then evaluateswhether to proceed to the next stage (1066), or repeat the above processof computing orientation vector candidates from another frame (1068).

If the detection values of one or more super-groups exceed a threshold,then the detector proceeds to the next stage. If not, the detectorgathers evidence of the orientation signal from another frame andreturns to the inter-frame coincidence matching process. Ultimately,when the detector finds sufficient evidence to proceed to the nextstage, it selects the super-group with the highest combined detectionvalue (1070), and sorts the blocks based on their correspondingdetection values (1072) to produce a ranked set of blocks for the nextstage (1074).

4.3 Estimating Translation Parameters

FIG. 17 is a flow diagram illustrating a method for estimatingtranslation parameters of the orientation signal, using informationgathered from the previous stages.

In this stage, the detector estimates translation parameters. Theseparameters indicate the starting point of a watermarked block in thespatial domain. The translation parameters, along with rotation, scaleand differential scale, form a complete 6D orientation vector. The 6Dvector enables the reader to extract luminance sample data inapproximately the same orientation as in the original watermarked image.

One approach is to use generalized match filtering to find thetranslation parameters that provide the best correlation. Anotherapproach is to continue to collect evidence about the orientation vectorcandidates, and provide a more comprehensive ranking of the orientationvectors based on all of the evidence gathered thus far. The followingparagraphs describe an example of this type of an approach.

To extract translation parameters, the detector proceeds as follows. Inthe multi-frame case, the detector selects the frame that produced 4Dorientation vectors with the highest detection values (1080). It thenprocesses the blocks 1082 in that frame in the order of their detectionvalue. For each block (1084), it applies the 4D vector to the luminancedata to generate rectified block data (1086). The detector then performsdual axis filtering (1088) and the window function (1090) on the data.Next, it performs an FFT (1092) on the image data to generate an arrayof Fourier data. To make correlation operations more efficient, thedetector buffers the fourier values at the orientation points (1094).

The detector applies a generalized match filter 1096 to correlate aphase specification of the orientation signal (1098) with thetransformed block data. The result of this process is a 2D array ofcorrelation values. The peaks in this array represent the translationparameters with the highest correlation. The detector selects the toppeaks and then applies a median filter to determine the center of eachof these peaks. The center of the peak has a corresponding correlationvalue and sub-pixel translation value. This process is one example ofgetting translation parameters by correlating the Fourier phasespecification of the orientation signal and the image data. Othermethods of phase locking the image data with a synchronization signallike the orientation signal may also be employed.

Depending on the implementation, the detector may have to resolveadditional ambiguities, such as rotation angle and flip ambiguity. Thedegree of ambiguity in the rotation angle depends on the nature of theorientation signal. If the orientation signal is octally symmetric(symmetric about horizontal, vertical and diagonal axes in the spatialfrequency domain), then the detector has to check each quadrant (0-90,90-180, 180-270, and 270-360 degrees) to find out which one the rotationangle resides in. Similarly, if the orientation signal is quadsymmetric, then the detector has to check two cases, 0-180 and 180-270.

The flip ambiguity may exist in some applications where the watermarkedimage can be flipped. To check for rotation and flip ambiguities, thedetector loops through each possible case, and performs the correlationoperation for each one (1100).

At the conclusion of the correlation process, the detector has produceda set of the top translation parameters with associated correlationvalues for each block. To gather additional evidence, the detectorgroups similar translation parameters from different blocks (1102),calculates a group detection value for each set of translationparameters 1104, and then ranks the top translation groups based ontheir corresponding group detection values 1106.

4.4 Refining Translation Parameters

Having gathered translation parameter estimates, the detector proceedsto refine these estimates. FIG. 18 is a flow diagram illustrating aprocess for refining orientation parameters. At this stage, the detectorprocess has gathered a set of the top translation parameter candidates1120 for a given frame 1122. The translation parameters provide anestimate of a reference point that locates the watermark, including boththe orientation and message components, in the image frame. In theimplementation depicted here, the translation parameters are representedas horizontal and vertical offsets from a reference point in the imageblock from which they were computed.

Recall that the detector has grouped translation parameters fromdifferent blocks based on their geometric proximity to each other. Eachpair of translation parameters in a group is associated with a block anda 4D vector (rotation, scale, and 2 differential scale parameters). Asshown in FIG. 18, the detector can now proceed to loop through eachgroup (1124), and through the blocks within each group (1126), to refinethe orientation parameters associated with each member of the groups.Alternatively, a simpler version of the detector may evaluate only thegroup with the highest detection value, or only selected blocks withinthat group.

Regardless of the number of candidates to be evaluated, the process ofrefining a given orientation vector candidate may be implemented in asimilar fashion. In the refining process, the detector uses a candidateorientation vector to define a mesh of sample blocks for furtheranalysis (1128). In one implementation, for example, the detector formsa mesh of 32 by 32 sample blocks centered around a seed block whoseupper right corner is located at the vertical and horizontal offsetspecified by the candidate translation parameters. The detector readssamples from each block using the orientation vector to extractluminance samples that approximate the original orientation of the hostimage at encoding time.

The detector steps through each block of samples (1130). For each block,it sets the orientation vector (1132), and then uses the orientationvector to check the validity of the watermark signal in the sampleblock. It assesses the validity of the watermark signal by calculating afigure of merit for the block (1134). To further refine the orientationparameters associated with each sample block, the detector adjustsselected parameters (e.g., vertical and horizontal translation) andre-calculates the figure of merit. As depicted in the inner loop in FIG.18 (block 1136 to 1132), the detector repeatedly adjusts the orientationvector and calculates the figure of merit in an attempt to find arefined orientation that yields a higher figure of merit.

The loop (1136) may be implemented by stepping through a predeterminedsequence of adjustments to parameters of the orientation vectors (e.g.,adding or subtracting small increments from the horizontal and verticaltranslation parameters). In this approach, the detector exits the loopafter stepping through the sequence of adjustments. Upon exiting, thedetector retains the orientation vector with the highest figure ofmerit.

There are a number of ways to calculate this figure of merit. One figureof merit is the degree of correlation between a known watermark signalattribute and a corresponding attribute in the signal suspected ofhaving a watermark. Another figure of merit is the strength of thewatermark signal (or one of its components) in the suspect signal. Forexample, a figure of merit may be based on a measure of the watermarkmessage signal strength and/or orientation pattern signal strength inthe signal, or in a part of the signal from which the detector extractsthe orientation parameters. The detector may computes a figure of meritbased the strength of the watermark signal in a sample block. It mayalso compute a figure of merit based on the percentage agreement betweenthe known bits of the message and the message bits extracted from thesample block.

When the figure of merit is computed based on a portion of the suspectsignal, the detector and reader can use the figure of merit to assessthe accuracy of the watermark signal detected and read from that portionof the signal. This approach enables the detector to assess the meritsof orientation parameters and to rank them based on their figure ofmerit. In addition, the reader can weight estimates of watermark messagevalues based on the figure of merit to recover a message more reliably.

The process of calculating a figure of merit depends on attributes thewatermark signal and how the embedder inserted it into the host signal.Consider an example where the watermark signal is added to the hostsignal. To calculate a figure of merit based on the strength of theorientation signal, the detector checks the value of each samplerelative to its neighbors, and compares the result with thecorresponding sample in a spatial domain version of the orientationsignal. When a sample's value is greater than its neighbors, then onewould expect that the corresponding orientation signal sample to bepositive. Conversely, when the sample's value is less than itsneighbors, then one would expect that the corresponding orientationsample to be negative. By comparing a sample's polarity relative to itsneighbors with the corresponding orientation sample's polarity, thedetector can assess the strength of the orientation signal in the sampleblock. In one implementation, the detector makes this polaritycomparison twice for each sample in an N by N block (e.g., N=32, 64,etc): once comparing each sample with its horizontally adjacentneighbors and then again comparing each sample with its verticallyadjacent neighbors. The detector performs this analysis on samples inthe mesh block after re-orienting the data to approximate the originalorientation of the host image at encoding time. The result of thisprocess is a number reflecting the portion of the total polaritycomparisons that yield a match.

To calculate a figure of merit based on known signature bits in amessage, the detector invokes the reader on the sample block, andprovides the orientation vector to enable the reader to extract codedmessage bits from the sample block. The detector compares the extractedmessage bits with the known bits to determine the extent to which theymatch. The result of this process is a percentage agreement numberreflecting the portion of the extracted message bits that match theknown bits. Together the test for the orientation signal and the messagesignal provide a figure of merit for the block.

As depicted in the loop from blocks 1138 to 1130, the detector mayrepeat the process of refining the orientation vector for each sampleblock around the seed block. In this case, the detector exits the loop(1138) after analyzing each of the sample blocks in the mesh definedpreviously (1128). In addition, the detector may repeat the analysis inthe loop through all blocks in a given group (1140), and in the loopthrough each group (1142).

After completing the analysis of the orientation vector candidates, thedetector proceeds to compute a combined detection value for the variouscandidates by compiling the results of the figure of merit calculations.It then proceeds to invoke the reader on the orientation vectorcandidates in the order of their detection values.

4.5 Reading the Watermark

FIG. 19 is a flow diagram illustrating a process for reading thewatermark message. Given an orientation vector and the correspondingimage data, the reader extracts the raw bits of a message from theimage. The reader may accumulate evidence of the raw bit values fromseveral different blocks. For example, in the process depicted in FIG.19, the reader uses refined orientation vectors for each block, andaccumulates evidence of the raw bit values extracted from the blocksassociated with the refined orientation vectors.

The reading process begins with a set of promising orientation vectorcandidates 1150 gathered from the detector. In each group of orientationvector candidates, there is a set of orientation vectors, eachcorresponding to a block in a given frame. The detector invokes thereader for one or more orientation vector groups whose detection valuesexceed a predetermined threshold. For each such group, the detectorloops over the blocks in the group (1152), and invokes the reader toextract evidence of the raw message bit values.

Recall that previous stages in the detector have refined orientationvectors to be used for the blocks of a group. When it invokes thereader, the detector provides the orientation vector as well as theimage block data (1154). The reader scans samples starting from alocation in a block specified by the translation parameters and usingthe other orientation parameters to approximate the original orientationof the image data (1156).

As described above, the embedder maps chips of the raw message bits toeach of the luminance samples in the original host image. Each sample,therefore, may provide an estimate of a chip's value. The readerreconstructs the value of the chip by first predicting the watermarksignal in the sample from the value of the sample relative to itsneighbors as described above (1158). If the deduced value appears valid,then the reader extracts the chip's value using the known value of thepseudo-random carrier signal for that sample and performing the inverseof the modulation function originally used to compute the watermarkinformation signal (1160). In particular, the reader performs anexclusive OR operation on the deduced value and the known carrier signalbit to get an estimate of the raw bit value. This estimate serves as anestimate for the raw bit value. The reader accumulates these estimatesfor each raw bit value (1162).

As noted above, the reader computes an estimate of the watermark signalby predicting the original, un-watermarked signal and deriving anestimate of the watermark signal based on the predicted signal and thewatermarked signal. It then computes an estimate of a raw bit valuebased on the value of the carrier signal, the assignment map that maps araw bit to the host image, and the relationship among the carrier signalvalue, the raw bit value, and the watermark signal value. In short, thereader reverses the embedding functions that modulate the message withthe carrier and apply the modulated carrier to the host signal. Usingthe predicted value of the original signal and an estimate of thewatermark signal, the reader reverses the embedding functions toestimate a value of the raw bit.

The reader loops over the candidate orientation vectors and associatedblocks, accumulating estimates for each raw bit value (1164). When theloop is complete, the reader calculates a final estimate value for eachraw bit from the estimates compiled for it. It then performs the inverseof the error correction coding operation on the final raw bit values(1166). Next, it performs a CRC to determine whether the read is valid.If no errors are detected, the read operation is complete and the readerreturns the message (1168).

However, if the read is invalid, then the detector may either attempt torefine the orientation vector data further, or start the detectionprocess with a new frame. Preferably, the detector should proceed torefine the orientation vector data when the combined detection value ofthe top candidates indicates that the current data is likely to containa strong watermark signal. In the process depicted in FIG. 19, forexample, the detector selects a processing path based on the combineddetection value (1170). The combined detection value may be calculatedin a variety of ways. One approach is to compute a combined detectionvalue based on the geometric coincidence of the top orientation vectorcandidates and a compilation of their figures of merit. The figure ofmerit may be computed as detailed earlier.

For cases where the read is invalid, the processing paths for theprocess depicted in FIG. 19 include: 1) refine the top orientationvectors in the spatial domain (1172); 2) invoke the translationestimator on the frame with the next best orientation vector candidates(1174); and 3) re-start the detection process on a new frame (assumingan implementation where more than one frame is available) (1176). Thesepaths are ranked in order from the highest detection value to thelowest. In the first case, the orientation vectors are the mostpromising. Thus, the detector re-invokes the reader on the samecandidates after refining them in the spatial domain (1178). In thesecond case, the orientation vectors are less promising, yet thedetection value indicates that it is still worthwhile to return to thetranslation estimation stage and continue from that point. Finally, inthe final case, the detection value indicates that the watermark signalis not strong enough to warrant further refinement. In this case, thedetector starts over with the next new frame of image data.

In each of the above cases, the detector continues to process the imagedata until it either makes a valid read, or has failed to make a validread after repeated passes through the available image data.

5.0 Operating Environment for Computer Implementations

FIG. 20 illustrates an example of a computer system that serves as anoperating environment for software implementations of the watermarkingsystems described above. The embedder and detector implementations areimplemented in C/C++ and are portable to many different computersystems. FIG. 20 generally depicts one such system.

The computer system shown in FIG. 20 includes a computer 1220, includinga processing unit 1221, a system memory 1222, and a system bus 1223 thatinterconnects various system components including the system memory tothe processing unit 1221.

The system bus may comprise any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using a bus architecture such as PCI, VESA, Microchannel(MCA), ISA and EISA, to name a few.

The system memory includes read only memory (ROM) 1224 and random accessmemory (RAM) 1225. A basic input/output system 1226 (BIOS), containingthe basic routines that help to transfer information between elementswithin the computer 1220, such as during start-up, is stored in ROM1224.

The computer 1220 further includes a hard disk drive 1227, a magneticdisk drive 1228, e.g., to read from or write to a removable disk 1229,and an optical disk drive 1230, e.g., for reading a CD-ROM or DVD disk1231 or to read from or write to other optical media. The hard diskdrive 1227, magnetic disk drive 1228, and optical disk drive 1230 areconnected to the system bus 1223 by a hard disk drive interface 1232, amagnetic disk drive interface 1233, and an optical drive interface 1234,respectively. The drives and their associated computer-readable mediaprovide nonvolatile storage of data, data structures,computer-executable instructions (program code such as dynamic linklibraries, and executable files), etc. for the computer 1220.

Although the description of computer-readable media above refers to ahard disk, a removable magnetic disk and an optical disk, it can alsoinclude other types of media that are readable by a computer, such asmagnetic cassettes, flash memory cards, digital video disks, and thelike.

A number of program modules may be stored in the drives and RAM 1225,including an operating system 1235, one or more application programs1236, other program modules 1237, and program data 1238.

A user may enter commands and information into the computer 1220 througha keyboard 1240 and pointing device, such as a mouse 1242. Other inputdevices may include a microphone, joystick, game pad, satellite dish,digital camera, scanner, or the like. A digital camera or scanner 43 maybe used to capture the target image for the detection process describedabove. The camera and scanner are each connected to the computer via astandard interface 44. Currently, there are digital cameras designed tointerface with a Universal Serial Bus (USB), Peripheral ComponentInterconnect (PCI), and parallel port interface. Two emerging standardperipheral interfaces for cameras include USB2 and 1394 (also known asfirewire and iLink).

Other input devices may be connected to the processing unit 1221 througha serial port interface 1246 or other port interfaces (e.g., a parallelport, game port or a universal serial bus (USB)) that are coupled to thesystem bus.

A monitor 1247 or other type of display device is also connected to thesystem bus 1223 via an interface, such as a video adapter 1248. Inaddition to the monitor, computers typically include other peripheraloutput devices (not shown), such as speakers and printers.

The computer 1220 operates in a networked environment using logicalconnections to one or more remote computers, such as a remote computer1249. The remote computer 1249 may be a server, a router, a peer deviceor other common network node, and typically includes many or all of theelements described relative to the computer 1220, although only a memorystorage device 1250 has been illustrated in FIG. 20. The logicalconnections depicted in FIG. 20 include a local area network (LAN) 1251and a wide area network (WAN) 1252. Such networking environments arecommonplace in offices, enterprise-wide computer networks, intranets andthe Internet.

When used in a LAN networking environment, the computer 1220 isconnected to the local network 1251 through a network interface oradapter 1253. When used in a WAN networking environment, the computer1220 typically includes a modem 1254 or other means for establishingcommunications over the wide area network 1252, such as the Internet.The modem 1254, which may be internal or external, is connected to thesystem bus 1223 via the serial port interface 1246.

In a networked environment, program modules depicted relative to thecomputer 1220, or portions of them, may be stored in the remote memorystorage device. The processes detailed above can be implemented in adistributed fashion, and as parallel processes. It will be appreciatedthat the network connections shown are exemplary and that other means ofestablishing a communications link between the computers may be used.

While the computer architecture depicted in FIG. 20 is similar totypical personal computer architectures, aspects of the invention may beimplemented in other computer architectures, such as hand-held computingdevices like Personal Digital Assistants, audio and/video players,network appliances, telephones, etc.

6.0 Improving Robustness to Attack

In some watermark applications, watermarks may be subject to varioustypes of attacks. These attacks may include attempting to remove thewatermark, attempting to copy it from one electronic signal or object toanother, and attempting to forge a watermark in a signal or physicalobject. These types of attacks may be more likely to occur in securityand copy protection applications. In these applications, the watermarkmay be used to convey rights (e.g., the right to copy or record) orauthenticate an electronic or physical object (such as an electronicobject like a video, audio or image file, or a tangible object such as adocument, identification card, ticket, etc.).

The following sections describe several countermeasures to attacks.These countermeasures are particularly directed to attacks where thewatermark is forged or copied into an electronic or physical object(object should be construed to cover an electronic and physical object,unless stated otherwise). In the case of a copy attack, the attackerattempts to estimate a watermark signal in an object expected to have avalid watermark. The attacker than inserts an estimate of the watermarksignal in another object. Forging a watermark refers to unauthorizedcreating and embedding a watermark signal into an object. While thecountermeasures discussed below are particularly useful in addressingthese types of attacks, the generally improve the robustness of awatermark in many applications and help counter other types of attacks.

Content and User Dependent Watermarks

One way to make the watermark more robust against malicious attack is tomake the watermark signal dependent on some attribute of the object intowhich it is to be embedded. For instance, the watermark key thatspecifies how or where the watermark is inserted in the object can bemake dependent on the object carrying the watermark. This approach makesit difficult to copy or forge the watermark in another object withouthaving the original object or knowing the attribute of the host that thewatermark is dependent on. The watermark key may be dependent on someattribute or attributes of the media signal in which the watermark is tobe embedded, such as the pixel values of an image or the audio samplevalues of an audio signal.

Alternatively, it may be dependent on some attribute that is uniquelyassigned to the object. For example, the object may be assigned with anidentifier that is uniquely associated with the owner or user of theobject, such as the owner of a watermarked identification card. Thisidentifier, being uniquely associated with a particular owner, may beused to generate a watermark key. If the watermark is copied or forgedinto an object used by another person, then that person will not be ableto provide a watermark key that enables a successful decode of thewatermark.

One way to implement a user dependent key is to use a person's name (orother unique attribute of that person) to create a watermark key. Thekey may be necessary to decode a scrambled watermark message, or tolocate the message in the watermarked object. The key may specify thetemporal or spatial location of the watermark. It may specify thetransform coefficients (e.g., frequency domain coefficients) where theobject has been altered to encode the watermark.

The name or other attribute of the person may be converted to awatermark key using a predetermined function. Alternatively, the name orother attribute may be used to look up a watermark key from a database(e.g., user's name or unique identifier number is linked to a specificwatermark key to be used to extract a watermark from the watermarkedobject). If the watermark decoder is unable to read the watermark usingthe key derived in this manner, then it indicates that the person is notthe valid owner or user of the watermarked object. This approach isparticularly suited for checking the validity of a photographicidentification document that has been watermarked using a user dependentkey.

Another approach to make the watermark more robust is to embed awatermark that is content dependent so that is difficult to forge orcopy from another object. Content dependent refers to a watermark signalthat is dependent on a host signal into which it is embedded. One way tomake the watermark content dependent is to transform at least a portionof the host signal to a transform coefficients in a selected transformdomain, and then alter one or more of the coefficients so that theycorrespond to a value of a desired symbol. The symbol is selected from apredetermined symbol alphabet in each symbol has one or more valuesassociated with it. The transform coefficients are modified so that theyhave a value corresponding to the desired symbol to be encoded. Thevalue to which the transform coefficients are modified is picked basedon the desired symbol to be encoded and the value associated with thatsymbol that requires the smallest change to the host signal. Thisapproach is a form of non-linear watermark encoding.

One example of the approach in the previous paragraph for image objectsis to perform a wavelet decomposition of at least a portion of the imageinto subbands. The pattern of coefficients in one or more of thesubbands is altered to correspond to a pattern associated with a symbolto be encoded. This process may be repeated to encode several symbols.In the decoder, the same wavelet decomposition is computed for thewatermarked image and the resulting patterns compared with the patternsin the symbol alphabet. For each pattern, the symbol with the closestmatching pattern is determined to be the one that has been encoded. Asimilar approach can be performed on other media types such as audio andvideo, using a wavelet transform or other types of transforms like anFFT, DCT, etc.

One particular technique for adapting the watermark signal to a hostimage signal in which it is embedded is to replace the magnitude (phaseis left in tact) of one set of Fourier coefficients,S₁=(C₁,C₂, . . . C_(n)), by an amount:(Avg₁+delta₁,Avg₂+delta₂, . . . Avg_(n)+delta_(n)),

and adjust a corresponding set of Fourier coefficients:S₁′=(C₁′,C₂′, . . . , C_(n)′) by an amount (Avg₁−delta₁,Avg₂−delta₂, . .. Avg_(n)−delta_(n)).Where Avg₁=(C₁′+C₁′)/2, Avg₂=(C₂+C₂′)/2, and Avg_(n)=(C_(n)+C_(n)′)/2.

The array of delta values can be any secret pattern, such as apseudorandom number generated by seeding a pseudorandom number generatorwith a secret key seed number. The key seed number itself may be derivedfrom features of the host signal. In fact, the mathematical relationshipbetween the coefficients in one set and the corresponding coefficientsin another set may be any of a variety of mathematical relationships.Changes to different coefficients in the set may be made according todifferent mathematical relationships. For example, the amount of change(e.g., delta in this example) may vary as a linear or non-linearrelationship of the frequency. Further, the approach may be repeated onadditional sets of coefficients S₂, S₃, . . . S_(n). Each set may usethe same or different key for the delta pattern. Different types ofmessages and authentication data may be encoded by applying the methodto certain ones of the sets of coefficients, and by varying the deltapatterns per each set of coefficients to correspond to desired messagesymbols to be encoded.

In one implementation, the coefficients in sets S₁ and S₁′ are selectedas shown in FIG. 21. Each coefficient in set S₁ is in a first quadrantof the Fourier transform domain of the host image, while correspondingcoefficients in set S₁′ are in a second quadrant. If these two quadrantswere folded together along a line separating the two quadrants, thecorresponding coefficients in S₁ and S₁′ would overlap.

After completing the watermark embedding process, the difference betweenthe corresponding sets of coefficients is a known pattern, (2delta₁,2delta₂, . . . 2delta_(n)).

At the detection stage, the detector computes the difference between atleast a subset of the corresponding sets of coefficients, possibly afterre-aligning the data to compensate for geometric distortion of thecaptured image relative to the image at the time of watermark embedding.The differences are then correlated with the intended pattern (delta₁,delta₂, . . . delta_(n)). If the correlation coefficient is less than athreshold, the captured image is deemed not to be authentic (e.g., aforgery or counterfeit).

By making the watermark content dependent using a non-linear encodingscheme, it is more difficult for the watermark attacker to forge or copythe watermark from a different object without detailed knowledge of theencoding process. Even if the encoding process is deduced in whole or inpart, the watermark message may be scrambled so that copying or forgingthe watermark will be ineffective. For example, the watermark messagemay be scrambled in a manner that is dependent on the name of the validowner of the object. Alternatively, the watermark message may bescrambled in a manner that is dependent on the signal content in whichthe watermark is embedded.

Yet another approach is to use content specific attributes of awatermark derived at the time of encoding to authenticate the watermarkin the decoding process. After the encoder has completed encoding awatermark into an object, it analyzes the watermarked object and derivesa characteristic or set of characteristics that describe attributes ofit. This attribute can be a characteristic signal manifested in atransform domain or in the native domain of the watermarked signal. Forexample, the attribute may be the location or location of frequencycoefficients that have signal energy above a given level. It may be anidentifier of a color and a corresponding range of watermark signalstrength in that color. For an image object, this characteristic may bemeasured by printing the watermarked image, scanning the image back to adigital domain, and then computing the characteristic. Next, thecharacteristic is stored in a database entry that is referenced via adatabase index in the watermark message. At decoding time, thecharacteristic is re-computed by scanning the watermarked image. Thecharacteristic computed at decoding time is then matched with thecharacteristic stored in the database to determine whether it issufficiently close to the stored characteristic. If so, it is deemedvalid; otherwise, it is rejected.

Post Processing to Reject Invalid Watermarks

Another way to counter watermark attacks is to use additional watermarkscreening stages to reject invalid watermarks. Since it is likely thatforged or copied watermarks are likely to be encoded with less strengthor at least differently than through the use of a valid watermarkencoder, the watermark detector may use this weakness of the forged orcopied watermark to reject it. The above sections described a techniquefor detecting watermarks in multiple stages. At different stages ofdetection, the detector computes detection values to assess whether aportion of a host signal is likely to contain a valid watermark. Ifthese detection values do not surpass predetermined thresholds, thewatermark may be rejected as being invalid.

Preferably, the watermark screening strategy is implemented in two ormore stages, with each stage being more discriminating in rejectedsignals. In particular, as the detector proceeds through each stage, thecandidate signals need to have detection values that surpass morestringent rejection tests (e.g., higher thresholds). Thus, even if aforged or copied watermark passes an initial stage, it is increasinglymore likely that it will get rejected in subsequent stages. Since thewatermark forger or copier does not have detailed knowledge of thedetector, he cannot be sure whether the forged or copied watermark willultimately pass the rejection tests.

Embedding Multiple Watermark Signals

Another way to counter attacks is to embed two or more differentwatermark signals. Each of the watermarks may be encoded with adifferent signal strength. The decoder can then determine whether thewatermark is valid by measuring the strength of the differentwatermarks. If a watermark component is not detected, or its detectedsignal strength falls below a desired threshold, the decoder rejects itas an invalid watermark.

Another variation of this approach is to encode watermarks at differentfrequencies (e.g., at high and low frequencies). One of the watermarkse.g., the one at the high frequency is less likely to survive through anattack. As such, the absence of this watermark at the decoding processindicates that the watermark is not valid. The watermark encoding may beimplemented by using watermark carrier signals at different frequenciesor by encoding the different watermark components at different frequencycoefficients.

Using Content Dependent Watermark Messages

Another way to counter attacks is to encode a content specific messageinto the watermark. At encoding time, a content specific message isderived from the host signal into which the watermark message is to beembedded. It is then included as at least part of the message that isencoded into the watermark. At decoding, the decoder derives the contentspecific message from the signal suspected of containing a validwatermark. It then decodes the watermark and compares the contentspecific message extracted from the watermark with the content specificmessage derived from the suspect signal. If the two are not within apredetermined tolerance of each other, then the decoder deems thewatermark to be invalid.

The content specific message may be computed in several ways. It shouldbe relatively insensitive to routine processing of the watermarkedsignal. For images, the content specific message should remainsubstantially unchanged through D to A and A to D conversion,compression/decompression, noise addition and small geometrictransformations. An image hash may be used as the content specificmessage, but it needs to be relatively insensitive to typicalmanipulation of the watermarked image. One form of semi-sensitive hashis a low pass filter of the image or a portion of it. The low passfiltered image can then be hashed into a number that is inserted intothe message. An example of a low pass filter is to use the mostsignificant bits of at least a selected part of the image.

Another content dependent message is a message derived based on salientfeatures in the content. For photographic images on identificationdocuments, the salient features may be the location of the eyes, nose ormouth in the photo of the document holder, or the relative location ofthese features. This location data (e.g., coordinates or hash of thecoordinates) can be converted into a numeric message, which is encodedinto the watermark embedded in the photographic image. If the watermarkis forged or copied into another image, then the content specificmessage in the watermark is unlikely to be within predeterminedtolerances of the content specific message computed from the image.

7.0 Identification Documents

Identification documents can be enhanced by using digital watermarks.This section describes processing of the ID document on the issuing andinspection sides to combat counterfeiting and forgery.

ID document Issuing Side

On the issuing side, a watermark encoder embeds a digital watermark in adigital photo (or other image to be placed on the ID document). Thewatermark carries a piece of multi-bit identifying data. This data canoptionally link to data on the card.

Preferably, the embedding process is tuned for survival through printingand scanning. A detailed description of such a process is providedabove.

The watermark encoder also embeds in a second version of the digitalphoto the same piece of identifying data. The second image may becompressed with a technique such as JPEG still image compression, oroptionally down-sampled to reduce storage requirements. The processes ofdown-sampling and compression may be used together or singly. Theembedding process in the second image may be less tuned to maintainimage quality and more tuned to survive manipulation such as imagecompression, down-sampling, or other transformation that it willundergo.

The first image is printed on the identification document (e.g., IDcard) using personalized printing methods that could range from ink-jetprinting to laser engraving.

The second image is encoded on the identification document in someportable storage media device or machine readable code such as amagnetic stripe, a 2D barcode, transistor, RF tag, magnetic ink, etc.

Alternatively, it is encoded in a database accessible to ID documentinspection systems. In such case, the ID document is linked to thesecond image via an identifier that serves as a database key to adatabase entry storing the second image. The inspection system may beequipped with the database as well as a database manager for retrievingimages via their respective keys. Alternatively, it may access thedatabase through a wire or wireless connection using standard networkcommunication protocols (e.g., via a network connection, which mayinclude a wire and/or wireless connection). The identifier may beembedded in the ID document in a watermark or some other storage deviceor machine readable code, such as the examples listed above.

ID Document Inspection Side

At inspection, a watermark decoder recovers data encoded in thewatermark from the printed image. Suitable implementations of awatermark decoder for detecting and reading a watermark from an imagecaptured from a document are described above.

The second version of the image is retrieved from the storage device ormachine readable code on the ID document or from remote storage, anddecompressed (if necessary). Next, the watermark decoder recovers thewatermark from this second image.

The pair of embedded identifiers in the first and second images providea machine-readable link between the two images (e.g., the photo on thedocument, and the second version encoded in the document or storedelsewhere). The images themselves provide a human viewable confirmationthat the image on the identification document has not been tamperedwith.

As an additional layer of security, the embedded data in the first andsecond images may be a piece of personal information, such as a numberuniquely associated with the valid bearer of the ID document. Thisenables the ID document system to create a verification triangle betweenthe bearer and the two images, as well as the one image to the other.

A possible variation is to store different but related identifiers inthe two images. One example is the start and end of the document number.Together, these related identifiers get compared to a correspondingidentifier on the ID document to check validity of the ID document. Tovalidate the ID document, the inspection device needs to recover theembedded data from the image printed on the document and the secondimage.

Another variation is to use some function f( ) where f(document number)yields some fairly unique x,y pair of numbers. The number x is encodedin a watermark in the printed image and the number y is encoded in awatermark in the image (encoded into the document or stored in adatabase). By employing a cryptographic function to generate the uniquenumber pair, it is possible to combat the threat of a counterfeiterlifting a watermark from the document and successfully reproducing it ona substitute photo.

Some examples of cipher algorithms for implementing a cryptographicfunction that generates a pair of numbers include RSA, DES, IDEA(International Data Encryption Algorithm), skipjack, discrete logsystems (e.g., El Gamal Cipher), elliptic curve systems, cellularautomata, etc.

The watermark may be used to detect block-level alterations in theprinted image. In particular, the watermark may be replicated in asimilar fashion in blocks of the image to be printed on the ID, and theimage kept separately (either encoded in the document or stored in adatabase). By comparing the watermark extracted from correspondingblocks in both images, the inspection device can detect alterations tothe printed image.

CONCLUDING REMARKS

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, applicants incorporate by reference the patents andpatent applications referenced above.

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patents/applications are also contemplated.

1. An object comprising: a first graphic including steganographic datahidden therein and provided on the object; a copy of the first graphicincluding steganographic data hidden therein and provided on the objector carried by the object, and the steganographic data hidden in thefirst graphic and the copy is related in a manner that enables validityof the object to be verified.
 2. The object of claim 1 wherein the copyof the first graphic is stored in a machine readable code on the object.3. The object of claim 2 wherein the copy of the first graphic iscompressed before being stored in the machine readable code.
 4. Theobject of claim 1 wherein the steganographic data hidden in the firstgraphic provided on the object includes a number that iscryptographically related to a number hidden in the copy of the firstgraphic.
 5. The object of claim 1 wherein the object compriseselectronic or integrated circuitry and the copy of the first graphic isstored in the electronic or integrated circuitry.
 6. An objectcomprising: a first graphic, wherein the first graphic comprises firststeganographic encoding therein; and electronic or integrated circuitry,wherein the electronic or integrated circuitry comprises a secondgraphic stored therein, wherein the second graphic comprises secondsteganographic encoding, and wherein the second encoding comprises aplural-bit identifier.
 7. The object of claim 6 wherein the firststeganographic encoding and the second steganographic encoding eachcomprise digital watermarking.
 8. The object of claim 6 wherein thefirst steganographic encoding and the second steganographic encoding arerelated in a manner that enables validity of the object to be verified.9. The object of claim 6 wherein the first steganographic encodingcomprises a plural-bit identifier, and wherein the plural-bitidentifiers comprise different information.
 10. The object of claim 9wherein the plural-bit identifiers comprise related information.
 11. Theobject of claim 6 wherein the second graphic is stored in a compressedform.
 12. The object of claim 6 wherein the first steganographicencoding comprises a plural-bit identifier, and wherein the plural-bitidentifiers are related through encryption in a manner that enablesvalidity of the object to be verified.
 13. The object of claim 6 whereinthe electronic or integrated circuitry comprises a wireless transmitter.14. The object of claim 6 wherein the electronic or integrated circuitrycomprises an RF tag.
 15. An object comprising: a first graphic providedon an object surface; and a storage device carried by the object,wherein the storage device comprises a second graphic stored therein,the first and second graphic are related graphics, and wherein thesecond graphic comprises first steganographic encoding including aplural-bit payload.
 16. The object of claim 15 wherein the storagedevice comprises electronic or integrated circuitry.
 17. The object ofclaim 15 wherein the first graphic comprises second steganographicencoding.
 18. The object of claim 17 wherein the first steganographicencoding and the second steganographic encoding each comprise digitalwatermarking.
 19. The object of claim 17 wherein the firststeganographic encoding and the second steganographic encoding arerelated in a manner that enables validity of the object to be verified.20. The object of claim 17 wherein the second steganographic encodingcomprises a plural-bit payload, and wherein the plural-bit payloadscomprise different information.
 21. The object of claim 1 wherein thesteganographic data hidden in the first graphic is hidden throughalterations to data representing the first graphic in a transformdomain.
 22. The object of claim 21 wherein the steganographic datahidden in the copy of the first graphic is hidden through alterations todata representing the copy of the first graphic in a transform domain.